Attachment 885646 Details for Bug 1086951 – engine.log file

engine.log file

engine.log (text/plain), 83.35 KB, created by Maurice James on 2014-04-12 03:19:20 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Maurice James

Created: 2014-04-12 03:19:20 UTC

Size: 83.35 KB

patch

obsolete

>2014-04-11 23:15:58,274 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:15:58,311 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) HostName = vhost3
>2014-04-11 23:15:58,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:15:58,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:15:58,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:15:58,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c5d6a7a
>2014-04-11 23:15:58,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) spmStart polling started: taskId = b1fb7824-9583-43b8-a4d6-f7e9a27f1321
>2014-04-11 23:16:00,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:16:00,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) spmStart polling ended: taskId = b1fb7824-9583-43b8-a4d6-f7e9a27f1321 task status = finished
>2014-04-11 23:16:00,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:16:00,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) spmStart polling ended, spm status: Free
>2014-04-11 23:16:00,545 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=b1fb7824-9583-43b8-a4d6-f7e9a27f1321), log id: 15b6cbb
>2014-04-11 23:16:00,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, HSMClearTaskVDSCommand, log id: 15b6cbb
>2014-04-11 23:16:00,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@40eef35b, log id: 3c5d6a7a
>2014-04-11 23:16:00,596 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-28) [7aafcab] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:16:00,640 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [7aafcab] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:16:10,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:16:10,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] HostName = vhost3
>2014-04-11 23:16:10,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:16:10,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:16:10,949 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:16:10,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4cbbdaf8
>2014-04-11 23:16:10,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] spmStart polling started: taskId = 0134d895-16f2-4b76-8192-47ae182b94f9
>2014-04-11 23:16:13,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:16:13,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] spmStart polling ended: taskId = 0134d895-16f2-4b76-8192-47ae182b94f9 task status = finished
>2014-04-11 23:16:13,109 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:16:13,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] spmStart polling ended, spm status: Free
>2014-04-11 23:16:13,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=0134d895-16f2-4b76-8192-47ae182b94f9), log id: 27fc8c91
>2014-04-11 23:16:13,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] FINISH, HSMClearTaskVDSCommand, log id: 27fc8c91
>2014-04-11 23:16:13,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [2dc742aa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34e1da67, log id: 4cbbdaf8
>2014-04-11 23:16:13,256 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [57c9bb50] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:16:13,299 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [57c9bb50] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:16:20,503 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [251c919c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host vhost3 to the Storage Domains Exports.
>2014-04-11 23:16:20,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-8) [251c919c] FINISH, ConnectStorageServerVDSCommand, return: {dced514e-9d49-4df5-b32e-2d6a27acac54=477}, log id: 273a0cb7
>2014-04-11 23:16:20,506 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [251c919c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection ashtistg01.suprtekstic.com:/storage/exports returned by VDSM was: Problem while trying to mount target
>2014-04-11 23:16:20,508 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-6-thread-8) [251c919c] The connection with details ashtistg01.suprtekstic.com:/storage/exports failed because of error code 477 and error message is: problem while trying to mount target
>2014-04-11 23:16:20,512 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-8) [251c919c] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-04-11 23:16:20,513 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [1001476d] ConnectDomainToStorage. After Connect all hosts to pool. Time:4/11/14 11:16 PM
>2014-04-11 23:16:23,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:16:23,477 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] HostName = vhost3
>2014-04-11 23:16:23,477 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:16:23,534 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) [62701136] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:16:23,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) [62701136] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:16:23,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44cc7c92
>2014-04-11 23:16:23,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] spmStart polling started: taskId = 3b300553-25ac-4d6c-a778-f579e2c5e359
>2014-04-11 23:16:26,844 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:16:26,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] spmStart polling ended: taskId = 3b300553-25ac-4d6c-a778-f579e2c5e359 task status = finished
>2014-04-11 23:16:26,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:16:26,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] spmStart polling ended, spm status: Free
>2014-04-11 23:16:26,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=3b300553-25ac-4d6c-a778-f579e2c5e359), log id: 6ccfd79a
>2014-04-11 23:16:26,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] FINISH, HSMClearTaskVDSCommand, log id: 6ccfd79a
>2014-04-11 23:16:26,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-83) [62701136] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1dfead34, log id: 44cc7c92
>2014-04-11 23:16:26,996 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-83) [3b1d404e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:16:27,050 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) [3b1d404e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:16:37,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:16:37,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) HostName = vhost3
>2014-04-11 23:16:37,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:16:37,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:16:37,319 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:16:37,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7db92cfd
>2014-04-11 23:16:37,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) spmStart polling started: taskId = 9c3df770-d2ba-47bc-86b4-73c4376ae747
>2014-04-11 23:16:39,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:16:39,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) spmStart polling ended: taskId = 9c3df770-d2ba-47bc-86b4-73c4376ae747 task status = finished
>2014-04-11 23:16:39,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:16:39,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) spmStart polling ended, spm status: Free
>2014-04-11 23:16:39,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-46) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=9c3df770-d2ba-47bc-86b4-73c4376ae747), log id: 6509ff4d
>2014-04-11 23:16:39,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, HSMClearTaskVDSCommand, log id: 6509ff4d
>2014-04-11 23:16:39,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d63f00e, log id: 7db92cfd
>2014-04-11 23:16:39,648 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-46) [7c253e69] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:16:39,692 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) [7c253e69] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:16:49,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:16:49,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] HostName = vhost3
>2014-04-11 23:16:49,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:16:49,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:16:49,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:16:49,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 473a4c42
>2014-04-11 23:16:50,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] spmStart polling started: taskId = ff8de269-e156-40b1-aee8-2e408d66332d
>2014-04-11 23:16:52,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:16:52,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] spmStart polling ended: taskId = ff8de269-e156-40b1-aee8-2e408d66332d task status = finished
>2014-04-11 23:16:52,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:16:52,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] spmStart polling ended, spm status: Free
>2014-04-11 23:16:52,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=ff8de269-e156-40b1-aee8-2e408d66332d), log id: 6624fd1b
>2014-04-11 23:16:52,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] FINISH, HSMClearTaskVDSCommand, log id: 6624fd1b
>2014-04-11 23:16:52,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [655ed227] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3edf0721, log id: 473a4c42
>2014-04-11 23:16:52,259 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-73) [244d27c6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:16:52,305 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [244d27c6] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:17:02,452 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:17:02,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) HostName = vhost3
>2014-04-11 23:17:02,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:17:02,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:17:02,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:17:02,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a265be2
>2014-04-11 23:17:02,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) spmStart polling started: taskId = c7aa7073-955e-4592-b31a-82ec35482d40
>2014-04-11 23:17:04,627 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:17:04,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) spmStart polling ended: taskId = c7aa7073-955e-4592-b31a-82ec35482d40 task status = finished
>2014-04-11 23:17:04,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:17:04,691 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) spmStart polling ended, spm status: Free
>2014-04-11 23:17:04,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-34) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=c7aa7073-955e-4592-b31a-82ec35482d40), log id: 23e706b0
>2014-04-11 23:17:04,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, HSMClearTaskVDSCommand, log id: 23e706b0
>2014-04-11 23:17:04,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@631ed000, log id: 6a265be2
>2014-04-11 23:17:04,777 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-34) [7be9e2d6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:04,822 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [7be9e2d6] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:17:14,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:17:15,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] HostName = vhost3
>2014-04-11 23:17:15,001 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:17:15,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [b180280] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:17:15,060 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [b180280] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:17:15,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2abb356
>2014-04-11 23:17:15,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] spmStart polling started: taskId = 8c7399eb-f318-4d58-ada4-40eff373d002
>2014-04-11 23:17:17,129 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:17:17,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] spmStart polling ended: taskId = 8c7399eb-f318-4d58-ada4-40eff373d002 task status = finished
>2014-04-11 23:17:17,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:17:17,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] spmStart polling ended, spm status: Free
>2014-04-11 23:17:17,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=8c7399eb-f318-4d58-ada4-40eff373d002), log id: 17b03c05
>2014-04-11 23:17:17,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] FINISH, HSMClearTaskVDSCommand, log id: 17b03c05
>2014-04-11 23:17:17,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-38) [b180280] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42207179, log id: 2abb356
>2014-04-11 23:17:17,279 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-38) [5ff9e0d9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:17,341 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [5ff9e0d9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:17:27,492 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:17:27,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] HostName = vhost3
>2014-04-11 23:17:27,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:17:27,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:17:27,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:17:27,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ca3e9f4
>2014-04-11 23:17:27,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] spmStart polling started: taskId = 6d82766f-c8c2-4c99-b99c-57649c7cb8b7
>2014-04-11 23:17:29,725 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:17:29,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] spmStart polling ended: taskId = 6d82766f-c8c2-4c99-b99c-57649c7cb8b7 task status = finished
>2014-04-11 23:17:29,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:17:29,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] spmStart polling ended, spm status: Free
>2014-04-11 23:17:29,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=6d82766f-c8c2-4c99-b99c-57649c7cb8b7), log id: 542828fa
>2014-04-11 23:17:29,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] FINISH, HSMClearTaskVDSCommand, log id: 542828fa
>2014-04-11 23:17:29,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) [6af82d9a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5e5f8abf, log id: 4ca3e9f4
>2014-04-11 23:17:29,875 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-81) [528e3200] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:29,933 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [528e3200] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:17:36,502 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue.
>2014-04-11 23:17:36,503 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-39) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue.
>2014-04-11 23:17:36,503 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-43) [9e93d75] Start running CanDoAction for command number 3/3 (Command type: ActivateVds)
>2014-04-11 23:17:36,506 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-39) [7d2715d1] Start running CanDoAction for command number 1/3 (Command type: ActivateVds)
>2014-04-11 23:17:36,505 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-42) [11098c88] Start running CanDoAction for command number 2/3 (Command type: ActivateVds)
>2014-04-11 23:17:36,509 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-43) [9e93d75] Lock Acquired to object EngineLock [exclusiveLocks= key: 6075a403-68b6-44d4-8cf9-9f7574a2cd61 value: VDS
>, sharedLocks= ]
>2014-04-11 23:17:36,510 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-39) [7d2715d1] Lock Acquired to object EngineLock [exclusiveLocks= key: ab667fbd-e3dc-4358-876c-9d0a2df14a44 value: VDS
>, sharedLocks= ]
>2014-04-11 23:17:36,511 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-42) [11098c88] Lock Acquired to object EngineLock [exclusiveLocks= key: cd40d6e4-99d5-4c73-a288-b39c254c76c4 value: VDS
>, sharedLocks= ]
>2014-04-11 23:17:36,547 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-43) [9e93d75] Finish handling CanDoAction for command number 3/3 (Command type: ActivateVds)
>2014-04-11 23:17:36,551 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-39) [7d2715d1] Finish handling CanDoAction for command number 1/3 (Command type: ActivateVds)
>2014-04-11 23:17:36,572 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-42) [11098c88] Finish handling CanDoAction for command number 2/3 (Command type: ActivateVds)
>2014-04-11 23:17:36,610 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: ab667fbd-e3dc-4358-876c-9d0a2df14a44 Type: VDS
>2014-04-11 23:17:36,611 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] Before acquiring lock in order to prevent monitoring for host vhost1 from data-center SuprTEK
>2014-04-11 23:17:36,612 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] Lock acquired, from now a monitoring of host will be skipped for host vhost1 from data-center SuprTEK
>2014-04-11 23:17:36,656 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] START, SetVdsStatusVDSCommand(HostName = vhost1, HostId = ab667fbd-e3dc-4358-876c-9d0a2df14a44, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 31481de5
>2014-04-11 23:17:36,662 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] FINISH, SetVdsStatusVDSCommand, log id: 31481de5
>2014-04-11 23:17:36,698 INFO  [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] START, ActivateVdsVDSCommand(HostName = vhost1, HostId = ab667fbd-e3dc-4358-876c-9d0a2df14a44), log id: 3aecc3ba
>2014-04-11 23:17:37,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] START, GetHardwareInfoVDSCommand(HostName = vhost1, HostId = ab667fbd-e3dc-4358-876c-9d0a2df14a44, vds=Host[vhost1]), log id: 391ddcbd
>2014-04-11 23:17:37,083 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (org.ovirt.thread.pool-6-thread-45) [7d2715d1] FINISH, GetHardwareInfoVDSCommand, log id: 391ddcbd
>2014-04-11 23:17:37,150 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (org.ovirt.thread.pool-6-thread-45) [4e3dbee0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: ab667fbd-e3dc-4358-876c-9d0a2df14a44 Type: VDS
>2014-04-11 23:17:37,189 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (org.ovirt.thread.pool-6-thread-45) [420797ac] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: ab667fbd-e3dc-4358-876c-9d0a2df14a44 Type: VDS
>2014-04-11 23:17:37,191 INFO  [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (org.ovirt.thread.pool-6-thread-45) [420797ac] FINISH, ActivateVdsVDSCommand, return: Host[vhost1], log id: 3aecc3ba
>2014-04-11 23:17:37,564 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [420797ac] Activate finished. Lock released. Monitoring can run now for host vhost1 from data-center SuprTEK
>2014-04-11 23:17:37,591 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [420797ac] Correlation ID: 7d2715d1, Job ID: ae5c6a1c-543a-461c-923d-81717119429a, Call Stack: null, Custom Event ID: -1, Message: Host vhost1 was activated by admin.
>2014-04-11 23:17:37,602 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [420797ac] Lock freed to object EngineLock [exclusiveLocks= key: ab667fbd-e3dc-4358-876c-9d0a2df14a44 value: VDS
>, sharedLocks= ]
>2014-04-11 23:17:37,628 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: cd40d6e4-99d5-4c73-a288-b39c254c76c4 Type: VDS
>2014-04-11 23:17:37,629 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] Before acquiring lock in order to prevent monitoring for host vhost2 from data-center SuprTEK
>2014-04-11 23:17:37,630 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] Lock acquired, from now a monitoring of host will be skipped for host vhost2 from data-center SuprTEK
>2014-04-11 23:17:37,668 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] START, SetVdsStatusVDSCommand(HostName = vhost2, HostId = cd40d6e4-99d5-4c73-a288-b39c254c76c4, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 587f2a63
>2014-04-11 23:17:37,674 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] FINISH, SetVdsStatusVDSCommand, log id: 587f2a63
>2014-04-11 23:17:37,708 INFO  [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] START, ActivateVdsVDSCommand(HostName = vhost2, HostId = cd40d6e4-99d5-4c73-a288-b39c254c76c4), log id: 1a0ceb17
>2014-04-11 23:17:38,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] START, GetHardwareInfoVDSCommand(HostName = vhost2, HostId = cd40d6e4-99d5-4c73-a288-b39c254c76c4, vds=Host[vhost2]), log id: 58015fa8
>2014-04-11 23:17:39,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (org.ovirt.thread.pool-6-thread-45) [11098c88] FINISH, GetHardwareInfoVDSCommand, log id: 58015fa8
>2014-04-11 23:17:39,249 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (org.ovirt.thread.pool-6-thread-45) [421c5757] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: cd40d6e4-99d5-4c73-a288-b39c254c76c4 Type: VDS
>2014-04-11 23:17:39,286 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (org.ovirt.thread.pool-6-thread-45) [422b2662] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: cd40d6e4-99d5-4c73-a288-b39c254c76c4 Type: VDS
>2014-04-11 23:17:39,288 INFO  [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (org.ovirt.thread.pool-6-thread-45) [422b2662] FINISH, ActivateVdsVDSCommand, return: Host[vhost2], log id: 1a0ceb17
>2014-04-11 23:17:39,669 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [422b2662] Activate finished. Lock released. Monitoring can run now for host vhost2 from data-center SuprTEK
>2014-04-11 23:17:39,677 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [422b2662] Correlation ID: 11098c88, Job ID: 4ddc4a2c-74dd-4b0d-bcc7-97b85bc68f23, Call Stack: null, Custom Event ID: -1, Message: Host vhost2 was activated by admin.
>2014-04-11 23:17:39,686 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [422b2662] Lock freed to object EngineLock [exclusiveLocks= key: cd40d6e4-99d5-4c73-a288-b39c254c76c4 value: VDS
>, sharedLocks= ]
>2014-04-11 23:17:39,711 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: 6075a403-68b6-44d4-8cf9-9f7574a2cd61 Type: VDS
>2014-04-11 23:17:39,712 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] Before acquiring lock in order to prevent monitoring for host vhost4 from data-center SuprTEK
>2014-04-11 23:17:39,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [3aed1419] START, GetHardwareInfoVDSCommand(HostName = vhost1, HostId = ab667fbd-e3dc-4358-876c-9d0a2df14a44, vds=Host[vhost1]), log id: 34ee32db
>2014-04-11 23:17:39,713 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] Lock acquired, from now a monitoring of host will be skipped for host vhost4 from data-center SuprTEK
>2014-04-11 23:17:39,754 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] START, SetVdsStatusVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 17f5e8be
>2014-04-11 23:17:39,761 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [3aed1419] FINISH, GetHardwareInfoVDSCommand, log id: 34ee32db
>2014-04-11 23:17:39,773 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] FINISH, SetVdsStatusVDSCommand, log id: 17f5e8be
>2014-04-11 23:17:39,810 INFO  [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] START, ActivateVdsVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61), log id: 5bacdc9c
>2014-04-11 23:17:39,838 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3aed1419] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host vhost1.
>2014-04-11 23:17:39,845 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-11) [4ab24ed7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:39,851 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [1c24a008] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:39,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [1c24a008] START, ConnectStorageServerVDSCommand(HostName = vhost1, HostId = ab667fbd-e3dc-4358-876c-9d0a2df14a44, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, storageType = NFS, connectionList = [{ id: 3a714537-238b-4fbf-8362-da2f922b6e5c, connection: ashtivh02.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 523bf150-41d4-4c9c-9e5b-95b4452d3697, connection: ashtivh03.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: dced514e-9d49-4df5-b32e-2d6a27acac54, connection: ashtistg01.suprtekstic.com:/storage/exports, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: deb06d0b-14c6-490e-8fbb-552861511109, connection: ashtivh04.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: eee1a342-05f6-4d3a-92fb-176201b5c73e, connection: ashtistg01.suprtekstic.com:/storage/infrastructure, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: f68d3d5c-3406-4c67-a6da-c642174a0760, connection: ashtivh01.suprtekstic.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: fd790dd2-b58f-406d-8fe0-fb951143f7ac, connection: ashtivh01.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5813084a
>2014-04-11 23:17:40,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:17:40,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) HostName = vhost3
>2014-04-11 23:17:40,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:17:40,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:17:40,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:17:40,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5cf43c48
>2014-04-11 23:17:40,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling started: taskId = c35ee8c1-dea0-4782-a444-88ff1ffd9d5c
>2014-04-11 23:17:41,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] START, GetHardwareInfoVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, vds=Host[vhost4]), log id: 58399431
>2014-04-11 23:17:41,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (org.ovirt.thread.pool-6-thread-45) [9e93d75] FINISH, GetHardwareInfoVDSCommand, log id: 58399431
>2014-04-11 23:17:41,343 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (org.ovirt.thread.pool-6-thread-45) [b65d840] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 6075a403-68b6-44d4-8cf9-9f7574a2cd61 Type: VDS
>2014-04-11 23:17:41,379 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (org.ovirt.thread.pool-6-thread-45) [5ca3a1e3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 6075a403-68b6-44d4-8cf9-9f7574a2cd61 Type: VDS
>2014-04-11 23:17:41,382 INFO  [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (org.ovirt.thread.pool-6-thread-45) [5ca3a1e3] FINISH, ActivateVdsVDSCommand, return: Host[vhost4], log id: 5bacdc9c
>2014-04-11 23:17:41,740 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [5ca3a1e3] Activate finished. Lock released. Monitoring can run now for host vhost4 from data-center SuprTEK
>2014-04-11 23:17:41,750 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [5ca3a1e3] Correlation ID: 9e93d75, Job ID: 40fa7b06-0122-4a4f-9df7-84a3c8566392, Call Stack: null, Custom Event ID: -1, Message: Host vhost4 was activated by admin.
>2014-04-11 23:17:41,758 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-45) [5ca3a1e3] Lock freed to object EngineLock [exclusiveLocks= key: 6075a403-68b6-44d4-8cf9-9f7574a2cd61 value: VDS
>, sharedLocks= ]
>2014-04-11 23:17:41,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [4ebe3aef] START, GetHardwareInfoVDSCommand(HostName = vhost2, HostId = cd40d6e4-99d5-4c73-a288-b39c254c76c4, vds=Host[vhost2]), log id: 4804d1c9
>2014-04-11 23:17:41,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [4ebe3aef] FINISH, GetHardwareInfoVDSCommand, log id: 4804d1c9
>2014-04-11 23:17:41,944 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [4ebe3aef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host vhost2.
>2014-04-11 23:17:41,953 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [7999e2eb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:41,958 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [3beea3d0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:42,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [3beea3d0] START, ConnectStorageServerVDSCommand(HostName = vhost2, HostId = cd40d6e4-99d5-4c73-a288-b39c254c76c4, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, storageType = NFS, connectionList = [{ id: 3a714537-238b-4fbf-8362-da2f922b6e5c, connection: ashtivh02.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 523bf150-41d4-4c9c-9e5b-95b4452d3697, connection: ashtivh03.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: dced514e-9d49-4df5-b32e-2d6a27acac54, connection: ashtistg01.suprtekstic.com:/storage/exports, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: deb06d0b-14c6-490e-8fbb-552861511109, connection: ashtivh04.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: eee1a342-05f6-4d3a-92fb-176201b5c73e, connection: ashtistg01.suprtekstic.com:/storage/infrastructure, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: f68d3d5c-3406-4c67-a6da-c642174a0760, connection: ashtivh01.suprtekstic.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: fd790dd2-b58f-406d-8fe0-fb951143f7ac, connection: ashtivh01.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12618780
>2014-04-11 23:17:42,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:17:42,345 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended: taskId = c35ee8c1-dea0-4782-a444-88ff1ffd9d5c task status = finished
>2014-04-11 23:17:42,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:17:42,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended, spm status: Free
>2014-04-11 23:17:42,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=c35ee8c1-dea0-4782-a444-88ff1ffd9d5c), log id: 6263697a
>2014-04-11 23:17:42,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, HSMClearTaskVDSCommand, log id: 6263697a
>2014-04-11 23:17:42,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31a33bbc, log id: 5cf43c48
>2014-04-11 23:17:42,512 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [53a6cd19] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:42,555 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) [53a6cd19] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:17:44,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GetHardwareInfoVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, vds=Host[vhost4]), log id: 2b229245
>2014-04-11 23:17:44,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GetHardwareInfoVDSCommand, log id: 2b229245
>2014-04-11 23:17:44,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GetHardwareInfoVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, vds=Host[vhost4]), log id: 31a251f1
>2014-04-11 23:17:44,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GetHardwareInfoVDSCommand, log id: 31a251f1
>2014-04-11 23:17:44,635 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host vhost4.
>2014-04-11 23:17:44,643 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-49) [394257aa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:44,649 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [26c0128f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:44,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [26c0128f] START, ConnectStorageServerVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, storageType = NFS, connectionList = [{ id: 3a714537-238b-4fbf-8362-da2f922b6e5c, connection: ashtivh02.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 523bf150-41d4-4c9c-9e5b-95b4452d3697, connection: ashtivh03.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: dced514e-9d49-4df5-b32e-2d6a27acac54, connection: ashtistg01.suprtekstic.com:/storage/exports, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: deb06d0b-14c6-490e-8fbb-552861511109, connection: ashtivh04.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: eee1a342-05f6-4d3a-92fb-176201b5c73e, connection: ashtistg01.suprtekstic.com:/storage/infrastructure, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: f68d3d5c-3406-4c67-a6da-c642174a0760, connection: ashtivh01.suprtekstic.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: fd790dd2-b58f-406d-8fe0-fb951143f7ac, connection: ashtivh01.suprtekstic.com:/var/lib/exports/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 71b25d03
>2014-04-11 23:17:52,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:17:52,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] HostName = vhost3
>2014-04-11 23:17:52,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:17:52,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:17:52,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:17:52,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7dba01b9
>2014-04-11 23:17:52,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] spmStart polling started: taskId = 1a9b3030-0318-4229-85d5-19d8f4ba8d53
>2014-04-11 23:17:54,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:17:54,886 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] spmStart polling ended: taskId = 1a9b3030-0318-4229-85d5-19d8f4ba8d53 task status = finished
>2014-04-11 23:17:54,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:17:54,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] spmStart polling ended, spm status: Free
>2014-04-11 23:17:54,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=1a9b3030-0318-4229-85d5-19d8f4ba8d53), log id: 2f7244f4
>2014-04-11 23:17:55,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] FINISH, HSMClearTaskVDSCommand, log id: 2f7244f4
>2014-04-11 23:17:55,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-29) [51a0092f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4241a35e, log id: 7dba01b9
>2014-04-11 23:17:55,035 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-29) [634c924d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:17:55,079 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [634c924d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:18:05,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:18:05,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) HostName = vhost3
>2014-04-11 23:18:05,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:18:05,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:18:05,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:18:05,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f331558
>2014-04-11 23:18:05,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling started: taskId = 5e2dea61-97ed-4abd-9dfa-f5e394797a2e
>2014-04-11 23:18:07,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:18:07,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling ended: taskId = 5e2dea61-97ed-4abd-9dfa-f5e394797a2e task status = finished
>2014-04-11 23:18:07,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:18:07,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling ended, spm status: Free
>2014-04-11 23:18:07,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-91) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=5e2dea61-97ed-4abd-9dfa-f5e394797a2e), log id: 25e143eb
>2014-04-11 23:18:07,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, HSMClearTaskVDSCommand, log id: 25e143eb
>2014-04-11 23:18:07,588 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5541668c, log id: 2f331558
>2014-04-11 23:18:07,591 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-91) [38b39d0e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:18:07,638 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [38b39d0e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:18:17,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:18:17,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] HostName = vhost3
>2014-04-11 23:18:17,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:18:17,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:18:17,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:18:17,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f59c48a
>2014-04-11 23:18:17,965 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] spmStart polling started: taskId = 76b3b284-3c24-45a3-bf1d-ea89c073c8dd
>2014-04-11 23:18:19,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:18:19,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] spmStart polling ended: taskId = 76b3b284-3c24-45a3-bf1d-ea89c073c8dd task status = finished
>2014-04-11 23:18:19,991 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:18:20,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] spmStart polling ended, spm status: Free
>2014-04-11 23:18:20,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=76b3b284-3c24-45a3-bf1d-ea89c073c8dd), log id: 1b2627fe
>2014-04-11 23:18:20,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] FINISH, HSMClearTaskVDSCommand, log id: 1b2627fe
>2014-04-11 23:18:20,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-10) [50bf3645] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d2c2854, log id: 4f59c48a
>2014-04-11 23:18:20,143 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-10) [3e0bb1c2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:18:20,188 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [3e0bb1c2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:18:25,797 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [26c0128f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host vhost4 to the Storage Domains Infrastructure,Exports.
>2014-04-11 23:18:25,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [26c0128f] FINISH, ConnectStorageServerVDSCommand, return: {f68d3d5c-3406-4c67-a6da-c642174a0760=0, 523bf150-41d4-4c9c-9e5b-95b4452d3697=0, 3a714537-238b-4fbf-8362-da2f922b6e5c=0, deb06d0b-14c6-490e-8fbb-552861511109=0, eee1a342-05f6-4d3a-92fb-176201b5c73e=477, dced514e-9d49-4df5-b32e-2d6a27acac54=477, fd790dd2-b58f-406d-8fe0-fb951143f7ac=0}, log id: 71b25d03
>2014-04-11 23:18:25,813 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [26c0128f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection ashtistg01.suprtekstic.com:/storage/infrastructure returned by VDSM was: Problem while trying to mount target
>2014-04-11 23:18:25,816 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (DefaultQuartzScheduler_Worker-49) [26c0128f] The connection with details ashtistg01.suprtekstic.com:/storage/infrastructure failed because of error code 477 and error message is: problem while trying to mount target
>2014-04-11 23:18:25,822 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [26c0128f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection ashtistg01.suprtekstic.com:/storage/exports returned by VDSM was: Problem while trying to mount target
>2014-04-11 23:18:25,825 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (DefaultQuartzScheduler_Worker-49) [26c0128f] The connection with details ashtistg01.suprtekstic.com:/storage/exports failed because of error code 477 and error message is: problem while trying to mount target
>2014-04-11 23:18:25,827 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [26c0128f] Host vhost4 storage connection was failed 
>2014-04-11 23:18:25,832 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [26c0128f] Correlation ID: 26c0128f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host vhost4 to Storage Servers
>2014-04-11 23:18:25,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) START, ConnectStoragePoolVDSCommand(HostName = vhost4, HostId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, vds_spm_id = 3, masterDomainId = a033c2ac-0d01-490c-9552-99ca53d6a64a, masterVersion = 38), log id: 3b1d2976
>2014-04-11 23:18:27,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) FINISH, ConnectStoragePoolVDSCommand, log id: 3b1d2976
>2014-04-11 23:18:28,565 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [26c0128f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host vhost3 from cluster STIC was chosen as a proxy to execute Status command on Host vhost4.
>2014-04-11 23:18:28,567 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) [26c0128f] Using Host vhost3 from cluster STIC as proxy to execute Status command on Host vhost4
>2014-04-11 23:18:28,602 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) [26c0128f] Executing <Status> Power Management command, Proxy Host:vhost3, Agent:ipmilan, Target Host:vhost4, Management IP:172.16.10.8, User:root, Options:lanplus
>2014-04-11 23:18:28,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [26c0128f] START, FenceVdsVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, targetVdsId = 6075a403-68b6-44d4-8cf9-9f7574a2cd61, action = Status, ip = 172.16.10.8, port = , type = ipmilan, user = root, password = ******, options = 'lanplus'), log id: 9937be7
>2014-04-11 23:18:30,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:18:30,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) HostName = vhost3
>2014-04-11 23:18:30,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:18:30,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:18:30,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:18:30,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2889e2f1
>2014-04-11 23:18:30,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) spmStart polling started: taskId = 0ffd13f1-4985-4125-ae47-0b7e2d63f7e2
>2014-04-11 23:18:32,540 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:18:32,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) spmStart polling ended: taskId = 0ffd13f1-4985-4125-ae47-0b7e2d63f7e2 task status = finished
>2014-04-11 23:18:32,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:18:32,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) spmStart polling ended, spm status: Free
>2014-04-11 23:18:32,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-41) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=0ffd13f1-4985-4125-ae47-0b7e2d63f7e2), log id: 461fe1f2
>2014-04-11 23:18:32,693 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, HSMClearTaskVDSCommand, log id: 461fe1f2
>2014-04-11 23:18:32,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34160bd5, log id: 2889e2f1
>2014-04-11 23:18:32,698 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-41) [1362e7aa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:18:32,744 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [1362e7aa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed
>2014-04-11 23:18:42,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-04-11 23:18:42,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-63) HostName = vhost3
>2014-04-11 23:18:42,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command HSMGetAllTasksStatusesVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-04-11 23:18:42,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) hostFromVds::selectedVds - vhost3, spmStatus Free, storage pool SuprTEK
>2014-04-11 23:18:42,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) starting spm on vds vhost3, storage pool SuprTEK, prevId -1, LVER -1
>2014-04-11 23:18:43,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) START, SpmStartVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, storagePoolId = a106ab81-9d5f-49c1-aeaf-832a137b708c, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1215b219
>2014-04-11 23:18:43,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling started: taskId = bf0edb34-d50e-4773-bec6-20391882a774
>2014-04-11 23:18:45,069 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) Failed in HSMGetTaskStatusVDS method
>2014-04-11 23:18:45,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended: taskId = bf0edb34-d50e-4773-bec6-20391882a774 task status = finished
>2014-04-11 23:18:45,073 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358
>2014-04-11 23:18:45,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended, spm status: Free
>2014-04-11 23:18:45,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) START, HSMClearTaskVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, taskId=bf0edb34-d50e-4773-bec6-20391882a774), log id: fc55a76
>2014-04-11 23:18:45,182 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, HSMClearTaskVDSCommand, log id: fc55a76
>2014-04-11 23:18:45,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@301bbcfb, log id: 1215b219
>2014-04-11 23:18:45,187 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-63) [6d141b69] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: a106ab81-9d5f-49c1-aeaf-832a137b708c Type: StoragePool
>2014-04-11 23:18:45,231 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) [6d141b69] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed

Actions: View

Attachments on bug 1086951: 885644 | 885645 | 885646