Attachment 856064 Details for Bug 1058300 – vdsm.log

vdsm.log

file_1058300.txt (text/plain), 98.54 KB, created by Andrew Lau on 2014-01-27 13:32:11 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Andrew Lau

Created: 2014-01-27 13:32:11 UTC

Size: 98.54 KB

patch

obsolete

>2014-01-28 00:13:07,539 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-30) org.ovirt.engine.core.common.errors.VDSError@8ed5beb
>2014-01-28 00:13:07,545 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-30) Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)
>	at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>
>2014-01-28 00:13:08,506 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e8469] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 700ba87b
>2014-01-28 00:13:08,568 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-85) [3e8469] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:08,571 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-85) [3e8469] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:08,573 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e8469] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@60c8ce99, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3b7826ab}, log id: 700ba87b
>2014-01-28 00:13:08,577 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-85) [3e8469] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:08,579 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-85) [3e8469] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:09,822 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) VM zcs1-melb-example-net 48ad952b-4457-4ae1-bf85-93f869382298 moved from Up --> Paused
>2014-01-28 00:13:13,069 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM zcs1-melb-example-net has paused due to unknown storage error.
>2014-01-28 00:13:13,071 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) VM ipa2-melb-example-net 546b9733-3453-4210-90fd-2b79cf08d53b moved from Up --> Paused
>2014-01-28 00:13:13,115 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ipa2-melb-example-net has paused due to unknown storage error.
>2014-01-28 00:13:13,122 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) VM rabbitmq1-melb-example-net 01d705e3-6b62-4796-b9e4-4de1c477401a moved from Up --> Paused
>2014-01-28 00:13:13,214 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM rabbitmq1-melb-example-net has paused due to unknown storage error.
>2014-01-28 00:13:13,666 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 1a7963ef
>2014-01-28 00:13:13,765 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-64) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:13,768 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-64) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:13,769 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@65461558, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b030e85b}, log id: 1a7963ef
>2014-01-28 00:13:13,772 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-64) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:13,774 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-64) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:16,754 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) VM proxy1-melb-example-net e85be056-7715-485c-9cb8-69f9980ed3a3 moved from Up --> Paused
>2014-01-28 00:13:16,792 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM proxy1-melb-example-net has paused due to unknown storage error.
>2014-01-28 00:13:16,813 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) VM foreman-melb-example-net bbbd4efa-022d-44d3-8274-a0506bda2d5e moved from Up --> Paused
>2014-01-28 00:13:16,862 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM foreman-melb-example-net has paused due to unknown storage error.
>2014-01-28 00:13:16,923 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-1) domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA in problem. vds: HV01
>2014-01-28 00:13:18,858 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 78a34316
>2014-01-28 00:13:18,932 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:18,940 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:18,941 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@afbb5bee, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c9be7f7d}, log id: 78a34316
>2014-01-28 00:13:18,945 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:18,946 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:19,981 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) VM ipa1-melb-example-net b3cde33c-f4c5-4b42-8e9d-820b80abc2fb moved from Up --> Paused
>2014-01-28 00:13:20,024 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ipa1-melb-example-net has paused due to unknown storage error.
>2014-01-28 00:13:24,028 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 43584725
>2014-01-28 00:13:24,078 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-90) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:24,088 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-90) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:24,096 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@faebdfe8, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8a32620b}, log id: 43584725
>2014-01-28 00:13:24,109 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-90) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:24,117 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-90) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:29,207 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 550ee429
>2014-01-28 00:13:31,132 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-36) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:31,142 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-36) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:31,150 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@433d2b37, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8dd6fefd}, log id: 550ee429
>2014-01-28 00:13:31,166 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-36) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:31,174 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-36) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:36,262 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 14ae4f5
>2014-01-28 00:13:36,321 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-98) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:36,330 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-98) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:36,338 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f2d2a39d, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b95ab35d}, log id: 14ae4f5
>2014-01-28 00:13:36,353 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-98) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:36,361 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-98) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:41,448 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 423c7290
>2014-01-28 00:13:41,618 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:41,621 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:41,623 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a4d57173, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2493fa33}, log id: 423c7290
>2014-01-28 00:13:41,627 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:41,629 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:46,710 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-99) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 498025fa
>2014-01-28 00:13:46,775 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-99) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:46,784 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-99) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:46,792 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1031fe37, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@19e5662}, log id: 498025fa
>2014-01-28 00:13:46,808 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-99) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:46,815 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-99) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:51,903 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 1844e9cd
>2014-01-28 00:13:51,974 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-44) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:51,983 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-44) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:51,991 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3939ccd4, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e5b03fee}, log id: 1844e9cd
>2014-01-28 00:13:52,007 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-44) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:52,015 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-44) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:57,104 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-23) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 7f557db1
>2014-01-28 00:13:57,174 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-23) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:57,177 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-23) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:13:57,185 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-23) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@65ea3612, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@acf0762e}, log id: 7f557db1
>2014-01-28 00:13:57,201 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-23) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:13:57,208 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-23) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:02,297 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 644361e5
>2014-01-28 00:14:02,375 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:02,385 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:02,393 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ecd78451, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@5a24a11f}, log id: 644361e5
>2014-01-28 00:14:02,408 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:02,416 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:07,502 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 2f1dc196
>2014-01-28 00:14:07,563 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-91) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:07,565 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-91) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:07,569 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b4af0c27, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e7ad95b2}, log id: 2f1dc196
>2014-01-28 00:14:07,585 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-91) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:07,592 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-91) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:07,641 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterTasksListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 7e395c08
>2014-01-28 00:14:07,679 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterTasksListVDSCommand, log id: 7e395c08
>2014-01-28 00:14:07,681 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-92) org.ovirt.engine.core.common.errors.VDSError@146001f
>2014-01-28 00:14:07,682 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-92) Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)
>	at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>
>2014-01-28 00:14:09,435 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-14) [6980b72e] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 Type: VM
>2014-01-28 00:14:09,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-14) [6980b72e] START, SetVmTicketVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3, ticket=cJRhRjxGzEou, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 3ed25dc2
>2014-01-28 00:14:09,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-14) [6980b72e] FINISH, SetVmTicketVDSCommand, log id: 3ed25dc2
>2014-01-28 00:14:09,611 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-14) [6980b72e] Correlation ID: 6980b72e, Call Stack: null, Custom Event ID: -1, Message: user admin@internal initiated console session for VM dev1-melb-example-net
>2014-01-28 00:14:12,680 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 1a018206
>2014-01-28 00:14:12,761 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:12,769 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:12,771 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8b68d31f, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8fcfdce8}, log id: 1a018206
>2014-01-28 00:14:12,775 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:12,777 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:17,863 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 6ec8bd13
>2014-01-28 00:14:17,915 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-50) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:17,917 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-50) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:17,919 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e8fb3859, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1b2f0743}, log id: 6ec8bd13
>2014-01-28 00:14:17,922 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-50) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:17,924 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-50) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:23,005 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 165ff525
>2014-01-28 00:14:28,365 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset
>2014-01-28 00:14:28,367 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, GlusterVolumesListVDSCommand, log id: 165ff525
>2014-01-28 00:14:28,377 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetStatsVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset
>2014-01-28 00:14:28,385 ERROR [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-11) Error while refreshing Gluster lightweight data of cluster Data_Hall_8!: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset (Failed with error VDS_NETWORK_ERROR and code 5022)
>	at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) [bll.jar:]
>	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterJob.runVdsCommand(GlusterJob.java:65) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.fetchVolumes(GlusterSyncJob.java:392) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.fetchVolumes(GlusterSyncJob.java:378) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshVolumeData(GlusterSyncJob.java:349) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshClusterData(GlusterSyncJob.java:105) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:84) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) [:1.7.0_51]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>
>2014-01-28 00:14:28,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [79ca67a2] Command SpmStatusVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-01-28 00:14:28,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) vds::refreshVdsStats Failed getVdsStats,  vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, error = VDSNetworkException: java.net.SocketException: Connection reset
>2014-01-28 00:14:28,453 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool
>2014-01-28 00:14:28,464 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Failed to refresh VDS , vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, VDS Network Error, continuing.
>java.net.SocketException: Connection reset
>2014-01-28 00:14:28,508 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [4ae9a898] Correlation ID: 4ae9a898, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NextDC_M1. Setting Data Center status to Non Responsive (On host HV01, Error: Network error during communication with the Host.).
>2014-01-28 00:14:28,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused
>2014-01-28 00:14:31,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@42aa46c6
>2014-01-28 00:14:31,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) HostName = HV01
>2014-01-28 00:14:31,581 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-01-28 00:14:31,655 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:537)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:335)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236)
>	at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host HV01 is initializing. Message: Recovering from crash or Initializing
>2014-01-28 00:14:31,682 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Failed to refresh VDS , vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, error = Recovering from crash or Initializing, continuing.
>2014-01-28 00:14:34,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) [221e4d0a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78b87d34
>2014-01-28 00:14:34,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) [221e4d0a] HostName = HV01
>2014-01-28 00:14:34,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) [221e4d0a] Command GetCapabilitiesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-01-28 00:14:37,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7b19947c
>2014-01-28 00:14:37,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) HostName = HV01
>2014-01-28 00:14:37,973 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-01-28 00:14:38,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
>
>2014-01-28 00:14:38,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] HostName = HV01
>2014-01-28 00:14:38,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] Command HSMGetAllTasksStatusesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
>2014-01-28 00:14:41,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] START, GetHardwareInfoVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]), log id: 1c3aeb76
>2014-01-28 00:14:41,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] FINISH, GetHardwareInfoVDSCommand, log id: 1c3aeb76
>2014-01-28 00:14:41,669 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] START, GlusterServersListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 21d18ebd
>2014-01-28 00:14:41,771 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] FINISH, GlusterServersListVDSCommand, return: [172.16.0.11:CONNECTED], log id: 21d18ebd
>2014-01-28 00:14:41,812 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [430df2fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host HV01.
>2014-01-28 00:14:41,837 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [430df2fc] VM dev1-melb-example-net 9c9befe0-9e17-462e-9bce-27ddd684a6d3 moved from Up --> Paused
>2014-01-28 00:14:41,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] START, FullListVdsCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01], vmIds=[9c9befe0-9e17-462e-9bce-27ddd684a6d3]), log id: 65966bac
>2014-01-28 00:14:41,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] FINISH, FullListVdsCommand, return: [Ljava.util.HashMap;@76771d4e, log id: 65966bac
>2014-01-28 00:14:41,868 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) [430df2fc] Cancelling the recovery from crash timer for VDS 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 because vds started initializing
>2014-01-28 00:14:41,870 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-24) [1f65ffc5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool
>2014-01-28 00:14:41,878 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool
>2014-01-28 00:14:41,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, ConnectStorageServerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = GLUSTERFS, connectionList = [{ id: 594fb154-104f-4248-b87b-4d01cf9b5325, connection: 172.16.1.5:/VM-DATA, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cc660c4
>2014-01-28 00:14:41,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, ConnectStorageServerVDSCommand, return: {594fb154-104f-4248-b87b-4d01cf9b5325=0}, log id: 6cc660c4
>2014-01-28 00:14:41,993 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] Host HV01 storage connection was succeeded 
>2014-01-28 00:14:42,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, ConnectStorageServerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: f414e026-440e-46ac-ac3f-213042ee2cce, connection: engine.melb.example.net:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5200630c
>2014-01-28 00:14:42,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, ConnectStorageServerVDSCommand, return: {f414e026-440e-46ac-ac3f-213042ee2cce=0}, log id: 5200630c
>2014-01-28 00:14:42,067 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] Host HV01 storage connection was succeeded 
>2014-01-28 00:14:42,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) START, ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1), log id: 16acf4c0
>2014-01-28 00:14:42,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value 
> StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d']]
>2014-01-28 00:14:42,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) HostName = HV01
>2014-01-28 00:14:42,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) Command ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d'
>2014-01-28 00:14:42,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) FINISH, ConnectStoragePoolVDSCommand, log id: 16acf4c0
>2014-01-28 00:14:42,596 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) Could not connect host HV01 to pool NextDC_M1, as the master domain is in inactive/unknown status - not failing the operation
>2014-01-28 00:14:42,821 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-24) [143a11c8] Failed to run Power Management command on Host HV01, no running proxy Host was found.
>2014-01-28 00:14:42,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, SetMOMPolicyParametersVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 765621a4
>2014-01-28 00:14:42,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 765621a4
>2014-01-28 00:14:42,951 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, GetGlusterHostUUIDVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 10bad2ae
>2014-01-28 00:14:42,998 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, GetGlusterHostUUIDVDSCommand, return: 1fba63f6-2109-40a1-bd33-e051fa939b81, log id: 10bad2ae
>2014-01-28 00:14:43,042 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [143a11c8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host HV01.There is no other host in the data center that can be used to test the power management settings.
>2014-01-28 00:14:43,051 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [143a11c8] Correlation ID: 1f65ffc5, Call Stack: null, Custom Event ID: -1, Message: Host HV01 power management was verified successfully.
>2014-01-28 00:14:43,136 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [b0595b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 Type: VDS
>2014-01-28 00:14:43,168 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [7c462c25] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 Type: VDS
>2014-01-28 00:14:43,179 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-44) domain 436c344c-bc57-441b-b311-c9595c6039e1:ISOS in problem. vds: HV01
>2014-01-28 00:14:43,593 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 522fec1f
>2014-01-28 00:14:43,656 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-9) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:43,666 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-9) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:43,674 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@28eabbd2, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@dd83881a}, log id: 522fec1f
>2014-01-28 00:14:43,690 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-9) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:43,698 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-9) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:46,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-16) START, FullListVdsCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01], vmIds=[e85be056-7715-485c-9cb8-69f9980ed3a3, 48ad952b-4457-4ae1-bf85-93f869382298, 546b9733-3453-4210-90fd-2b79cf08d53b, bbbd4efa-022d-44d3-8274-a0506bda2d5e, b3cde33c-f4c5-4b42-8e9d-820b80abc2fb, 01d705e3-6b62-4796-b9e4-4de1c477401a]), log id: 74850c6b
>2014-01-28 00:14:46,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FullListVdsCommand, return: [Ljava.util.HashMap;@4af28090, log id: 74850c6b
>2014-01-28 00:14:46,358 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Received a bridge Device without an address when processing VM 48ad952b-4457-4ae1-bf85-93f869382298 devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:19:42:e0, linkActive=true, type=interface, deviceId=3200b542-606b-4e19-a123-9082787ca210, filter=vdsm-no-mac-spoofing, network=r2o_net}
>2014-01-28 00:14:46,374 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Received a bridge Device without an address when processing VM b3cde33c-f4c5-4b42-8e9d-820b80abc2fb devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:19:42:e1, linkActive=true, type=interface, deviceId=e80486d7-7833-477b-829e-4198688db798, filter=vdsm-no-mac-spoofing, network=r2o_net}
>2014-01-28 00:14:46,392 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Received a bridge Device without an address when processing VM 546b9733-3453-4210-90fd-2b79cf08d53b devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:19:42:1c, linkActive=true, type=interface, deviceId=7aa48c2d-acdf-41d4-972f-05cd92c5892a, filter=vdsm-no-mac-spoofing, network=r2o_net}
>2014-01-28 00:14:48,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] hostFromVds::selectedVds - HV01, spmStatus Unknown_Pool, storage pool NextDC_M1
>2014-01-28 00:14:48,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] START, ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1), log id: 5c03afee
>2014-01-28 00:14:48,791 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3ae772d6
>2014-01-28 00:14:48,867 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-22) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:48,869 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-22) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:48,871 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b22933bc, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@737c1ab}, log id: 3ae772d6
>2014-01-28 00:14:48,874 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-22) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:48,876 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-22) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:49,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value 
> StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d']]
>2014-01-28 00:14:49,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] HostName = HV01
>2014-01-28 00:14:49,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] Command ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d'
>2014-01-28 00:14:49,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] FINISH, ConnectStoragePoolVDSCommand, log id: 5c03afee
>2014-01-28 00:14:49,209 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-01-28 00:14:49,214 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d'
>2014-01-28 00:14:49,407 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: ff3a0c94-11dc-446d-b093-5f68ad81520d Type: Storage
>2014-01-28 00:14:49,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] START, ResetIrsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, vdsId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, ignoreStopFailed = true), log id: 2176ee7f
>2014-01-28 00:14:49,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] START, SpmStopVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3), log id: 1a389302
>2014-01-28 00:14:49,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
>
>2014-01-28 00:14:49,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] HostName = HV01
>2014-01-28 00:14:49,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Command HSMGetAllTasksStatusesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
>2014-01-28 00:14:49,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] FINISH, SpmStopVDSCommand, log id: 1a389302
>2014-01-28 00:14:49,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] FINISH, ResetIrsVDSCommand, log id: 2176ee7f
>2014-01-28 00:14:49,622 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Correlation ID: 156ff99d, Job ID: 20100c6d-068c-4445-9d35-91a669e15dc5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center NextDC_M1.
>2014-01-28 00:14:49,638 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-39) [66f2577c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue
>2014-01-28 00:14:53,965 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 25ba21d5
>2014-01-28 00:14:54,023 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:54,027 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:54,029 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@71e5a6ba, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c6be54ea}, log id: 25ba21d5
>2014-01-28 00:14:54,034 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:54,037 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:59,120 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) [2697e1d7] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 35ef9d
>2014-01-28 00:14:59,206 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:59,210 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:14:59,211 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) [2697e1d7] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@21989f42, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@eaa0a25}, log id: 35ef9d
>2014-01-28 00:14:59,215 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:59,216 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:14:59,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) hostFromVds::selectedVds - HV01, spmStatus Unknown_Pool, storage pool NextDC_M1
>2014-01-28 00:14:59,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) START, ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1), log id: 52216d1c
>2014-01-28 00:15:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) Autorecovering 1 storage domains
>2014-01-28 00:15:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) Autorecovering storage domains id: ff3a0c94-11dc-446d-b093-5f68ad81520d
>2014-01-28 00:15:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-41) [5e0d5090] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: ff3a0c94-11dc-446d-b093-5f68ad81520d Type: Storage
>2014-01-28 00:15:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-41) [5e0d5090] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/28/14 12:15 AM
>2014-01-28 00:15:00,128 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-43) [2200647a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-01-28 00:15:00,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [2200647a] START, ConnectStorageServerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 594fb154-104f-4248-b87b-4d01cf9b5325, connection: 172.16.1.5:/VM-DATA, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50796f8b
>2014-01-28 00:15:00,557 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [2200647a] FINISH, ConnectStorageServerVDSCommand, return: {594fb154-104f-4248-b87b-4d01cf9b5325=0}, log id: 50796f8b
>2014-01-28 00:15:00,571 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-41) [5e0d5090] ConnectDomainToStorage. After Connect all hosts to pool. Time:1/28/14 12:15 AM
>2014-01-28 00:15:03,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, ConnectStoragePoolVDSCommand, log id: 52216d1c
>2014-01-28 00:15:03,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) hostFromVds::Connected host to pool - selectedVds - HV01, spmStatus Free, storage pool NextDC_M1
>2014-01-28 00:15:04,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) SpmStatus on vds 723cdb54-f44c-4ffd-b1bf-fde7ec606d92: Free
>2014-01-28 00:15:04,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) starting spm on vds HV01, storage pool NextDC_M1, prevId 1, LVER 1
>2014-01-28 00:15:04,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) START, SpmStartVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, prevId=1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a177e76
>2014-01-28 00:15:04,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling started: taskId = 4155532d-c8c0-4274-8551-25b0193797b3
>2014-01-28 00:15:04,348 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 48f93b85
>2014-01-28 00:15:04,469 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:04,480 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:04,490 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@689bbf9, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@df26eff8}, log id: 48f93b85
>2014-01-28 00:15:04,510 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:04,519 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:07,760 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) START, GlusterTasksListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 39bf287a
>2014-01-28 00:15:07,844 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, GlusterTasksListVDSCommand, log id: 39bf287a
>2014-01-28 00:15:07,850 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-56) org.ovirt.engine.core.common.errors.VDSError@24c8951f
>2014-01-28 00:15:07,856 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-56) Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)
>	at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>
>2014-01-28 00:15:09,606 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 4e28d6be
>2014-01-28 00:15:09,656 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-13) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:09,666 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-13) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:09,674 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fc1070e2, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1e780c70}, log id: 4e28d6be
>2014-01-28 00:15:09,689 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-13) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:09,696 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-13) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:14,788 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3c6fdaf6
>2014-01-28 00:15:14,875 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:14,884 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:14,893 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fd92fde5, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7f2f8396}, log id: 3c6fdaf6
>2014-01-28 00:15:14,909 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-61) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:14,917 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-61) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:20,011 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) [44f165b7] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3b1eb02c
>2014-01-28 00:15:20,076 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-69) [44f165b7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:20,086 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-69) [44f165b7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:20,095 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) [44f165b7] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@81f1741c, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@dd5cefb4}, log id: 3b1eb02c
>2014-01-28 00:15:20,111 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-69) [44f165b7] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:20,118 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-69) [44f165b7] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:22,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling ended: taskId = 4155532d-c8c0-4274-8551-25b0193797b3 task status = finished
>2014-01-28 00:15:22,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling ended, spm status: SPM
>2014-01-28 00:15:22,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-91) START, HSMClearTaskVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, taskId=4155532d-c8c0-4274-8551-25b0193797b3), log id: 18beda7d
>2014-01-28 00:15:22,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, HSMClearTaskVDSCommand, log id: 18beda7d
>2014-01-28 00:15:22,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71d4b4c7, log id: 5a177e76
>2014-01-28 00:15:22,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) Initialize Irs proxy from vds: 172.16.0.11
>2014-01-28 00:15:22,800 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host HV01 (Address: 172.16.0.11).
>2014-01-28 00:15:25,210 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [6be25d2] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: fe55cf7
>2014-01-28 00:15:25,269 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-59) [6be25d2] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:25,273 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-59) [6be25d2] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:25,276 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [6be25d2] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e97bf33a, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ef3e059e}, log id: fe55cf7
>2014-01-28 00:15:25,281 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-59) [6be25d2] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:25,283 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-59) [6be25d2] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:30,365 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 18f6c356
>2014-01-28 00:15:30,456 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-5) [195dc414] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:30,466 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-5) [195dc414] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:30,475 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a3d5ce6d, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@57b0a1e9}, log id: 18f6c356
>2014-01-28 00:15:30,492 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-5) [195dc414] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:30,500 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-5) [195dc414] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:35,151 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) Storage Domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA was reported by Host HV01 as Active in Pool 5849b030-626e-47cb-ad90-3ce782d831b3, moving to active status
>2014-01-28 00:15:35,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA recovered from problem. vds: HV01
>2014-01-28 00:15:35,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-01-28 00:15:35,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain 436c344c-bc57-441b-b311-c9595c6039e1:ISOS recovered from problem. vds: HV01
>2014-01-28 00:15:35,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain 436c344c-bc57-441b-b311-c9595c6039e1:ISOS has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-01-28 00:15:35,592 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 60c6e137
>2014-01-28 00:15:35,644 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-96) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:35,646 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-96) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:35,648 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@126e824a, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a9d97791}, log id: 60c6e137
>2014-01-28 00:15:35,651 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-96) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:35,652 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-96) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:40,743 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) [4c514a31] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 698797b8
>2014-01-28 00:15:40,816 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [4c514a31] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:40,825 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [4c514a31] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:40,833 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) [4c514a31] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9f4386aa, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2e15b648}, log id: 698797b8
>2014-01-28 00:15:40,848 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-53) [4c514a31] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:40,855 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-53) [4c514a31] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:45,947 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 5c40bd0b
>2014-01-28 00:15:46,042 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-84) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:46,045 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-84) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:46,047 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b2bad5a7, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@114ae154}, log id: 5c40bd0b
>2014-01-28 00:15:46,051 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-84) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:46,053 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-84) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:51,134 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 647a7e41
>2014-01-28 00:15:51,186 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:51,189 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:51,190 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e9746c6d, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4adeb311}, log id: 647a7e41
>2014-01-28 00:15:51,193 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:51,194 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:56,278 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 4c492531
>2014-01-28 00:15:56,330 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-72) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:56,332 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-72) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:15:56,334 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@572dc7be, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@98a37c44}, log id: 4c492531
>2014-01-28 00:15:56,337 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-72) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:56,340 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-72) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:15:58,015 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-3) [6aad9c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 value: VM
>, sharedLocks= ]
>2014-01-28 00:15:58,090 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Running command: RunVmCommand internal: false. Entities affected :  ID: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 Type: VM
>2014-01-28 00:15:58,169 INFO  [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] START, ResumeVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3), log id: 5c1bea93
>2014-01-28 00:15:58,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] START, ResumeBrokerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3), log id: 57a2216
>2014-01-28 00:15:58,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Failed in ResumeBrokerVDS method
>2014-01-28 00:15:58,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.OneVmReturnForXmlRpc@5d0d34f7
>2014-01-28 00:15:58,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] HostName = HV01
>2014-01-28 00:15:58,239 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Command ResumeBrokerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to ResumeBrokerVDS, error = Unexpected exception, code = 16
>2014-01-28 00:15:58,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] FINISH, ResumeBrokerVDSCommand, log id: 57a2216
>2014-01-28 00:15:58,242 ERROR [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] VDS::pause Failed resume vm 9c9befe0-9e17-462e-9bce-27ddd684a6d3 in vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ResumeBrokerVDS, error = Unexpected exception, code = 16
>2014-01-28 00:15:58,244 INFO  [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] FINISH, ResumeVDSCommand, return: Unknown, log id: 5c1bea93
>2014-01-28 00:15:58,246 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Lock freed to object EngineLock [exclusiveLocks= key: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 value: VM
>, sharedLocks= ]
>2014-01-28 00:15:58,254 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Command org.ovirt.engine.core.bll.RunVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ResumeBrokerVDS, error = Unexpected exception, code = 16 (Failed with error unexpected and code 16)
>2014-01-28 00:15:58,279 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Correlation ID: 6aad9c7, Job ID: e58fee76-8475-4492-a7f3-74c77e3956cc, Call Stack: null, Custom Event ID: -1, Message: Failed to resume VM dev1-melb-example-net (Host: HV01, User: admin@internal).
>2014-01-28 00:16:01,430 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-67) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 159c155d
>2014-01-28 00:16:01,497 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-67) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:16:01,500 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-67) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:16:01,502 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a560d359, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe3b4ddd}, log id: 159c155d
>2014-01-28 00:16:01,508 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-67) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:16:01,510 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-67) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:16:06,594 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 4cd27c70
>2014-01-28 00:16:06,644 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-19) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:16:06,646 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-19) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:16:06,647 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c768bcd7, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@591f9b20}, log id: 4cd27c70
>2014-01-28 00:16:06,651 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-19) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:16:06,652 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-19) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:16:07,964 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] START, GlusterTasksListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: a023379
>2014-01-28 00:16:08,007 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] FINISH, GlusterTasksListVDSCommand, log id: a023379
>2014-01-28 00:16:08,009 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-6) [4ae9a898] org.ovirt.engine.core.common.errors.VDSError@74067b75
>2014-01-28 00:16:08,010 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-6) [4ae9a898] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)
>	at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:]
>	at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>
>2014-01-28 00:16:11,734 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3767e8ac
>2014-01-28 00:16:11,800 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:16:11,806 WARN  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95
>2014-01-28 00:16:11,808 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f5fe2e7b, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b70ce26a}, log id: 3767e8ac
>2014-01-28 00:16:11,818 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>2014-01-28 00:16:11,826 WARN  [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point.
>^C

Actions: View

Attachments on bug 1058300: 856063 | 856064 | 858008 | 858009 | 858010 | 867646 | 867647 | 884735 | 909506