Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 856064 Details for
Bug 1058300
VMs do not resume after paused state and storage connection to a gluster domain (they will also fail to be manually resumed)
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm.log
file_1058300.txt (text/plain), 98.54 KB, created by
Andrew Lau
on 2014-01-27 13:32:11 UTC
(
hide
)
Description:
vdsm.log
Filename:
MIME Type:
Creator:
Andrew Lau
Created:
2014-01-27 13:32:11 UTC
Size:
98.54 KB
patch
obsolete
>2014-01-28 00:13:07,539 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-30) org.ovirt.engine.core.common.errors.VDSError@8ed5beb >2014-01-28 00:13:07,545 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-30) Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed >error: tasks is not a valid status option >Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool] >return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161) > at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-01-28 00:13:08,506 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e8469] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 700ba87b >2014-01-28 00:13:08,568 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-85) [3e8469] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:08,571 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-85) [3e8469] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:08,573 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e8469] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@60c8ce99, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3b7826ab}, log id: 700ba87b >2014-01-28 00:13:08,577 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-85) [3e8469] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:08,579 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-85) [3e8469] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:09,822 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) VM zcs1-melb-example-net 48ad952b-4457-4ae1-bf85-93f869382298 moved from Up --> Paused >2014-01-28 00:13:13,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM zcs1-melb-example-net has paused due to unknown storage error. >2014-01-28 00:13:13,071 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) VM ipa2-melb-example-net 546b9733-3453-4210-90fd-2b79cf08d53b moved from Up --> Paused >2014-01-28 00:13:13,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ipa2-melb-example-net has paused due to unknown storage error. >2014-01-28 00:13:13,122 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) VM rabbitmq1-melb-example-net 01d705e3-6b62-4796-b9e4-4de1c477401a moved from Up --> Paused >2014-01-28 00:13:13,214 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM rabbitmq1-melb-example-net has paused due to unknown storage error. >2014-01-28 00:13:13,666 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 1a7963ef >2014-01-28 00:13:13,765 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-64) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:13,768 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-64) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:13,769 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@65461558, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b030e85b}, log id: 1a7963ef >2014-01-28 00:13:13,772 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-64) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:13,774 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-64) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:16,754 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) VM proxy1-melb-example-net e85be056-7715-485c-9cb8-69f9980ed3a3 moved from Up --> Paused >2014-01-28 00:13:16,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM proxy1-melb-example-net has paused due to unknown storage error. >2014-01-28 00:13:16,813 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) VM foreman-melb-example-net bbbd4efa-022d-44d3-8274-a0506bda2d5e moved from Up --> Paused >2014-01-28 00:13:16,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM foreman-melb-example-net has paused due to unknown storage error. >2014-01-28 00:13:16,923 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-1) domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA in problem. vds: HV01 >2014-01-28 00:13:18,858 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 78a34316 >2014-01-28 00:13:18,932 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:18,940 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:18,941 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@afbb5bee, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c9be7f7d}, log id: 78a34316 >2014-01-28 00:13:18,945 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:18,946 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:19,981 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) VM ipa1-melb-example-net b3cde33c-f4c5-4b42-8e9d-820b80abc2fb moved from Up --> Paused >2014-01-28 00:13:20,024 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ipa1-melb-example-net has paused due to unknown storage error. >2014-01-28 00:13:24,028 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 43584725 >2014-01-28 00:13:24,078 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-90) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:24,088 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-90) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:24,096 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@faebdfe8, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8a32620b}, log id: 43584725 >2014-01-28 00:13:24,109 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-90) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:24,117 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-90) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:29,207 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 550ee429 >2014-01-28 00:13:31,132 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-36) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:31,142 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-36) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:31,150 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@433d2b37, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8dd6fefd}, log id: 550ee429 >2014-01-28 00:13:31,166 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-36) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:31,174 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-36) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:36,262 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 14ae4f5 >2014-01-28 00:13:36,321 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-98) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:36,330 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-98) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:36,338 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f2d2a39d, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b95ab35d}, log id: 14ae4f5 >2014-01-28 00:13:36,353 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-98) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:36,361 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-98) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:41,448 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 423c7290 >2014-01-28 00:13:41,618 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:41,621 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:41,623 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a4d57173, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2493fa33}, log id: 423c7290 >2014-01-28 00:13:41,627 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:41,629 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:46,710 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-99) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 498025fa >2014-01-28 00:13:46,775 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-99) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:46,784 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-99) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:46,792 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1031fe37, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@19e5662}, log id: 498025fa >2014-01-28 00:13:46,808 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-99) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:46,815 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-99) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:51,903 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 1844e9cd >2014-01-28 00:13:51,974 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-44) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:51,983 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-44) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:51,991 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3939ccd4, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e5b03fee}, log id: 1844e9cd >2014-01-28 00:13:52,007 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-44) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:52,015 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-44) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:57,104 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-23) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 7f557db1 >2014-01-28 00:13:57,174 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-23) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:57,177 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-23) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:13:57,185 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-23) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@65ea3612, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@acf0762e}, log id: 7f557db1 >2014-01-28 00:13:57,201 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-23) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:13:57,208 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-23) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:02,297 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 644361e5 >2014-01-28 00:14:02,375 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:02,385 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:02,393 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ecd78451, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@5a24a11f}, log id: 644361e5 >2014-01-28 00:14:02,408 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:02,416 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:07,502 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 2f1dc196 >2014-01-28 00:14:07,563 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-91) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:07,565 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-91) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:07,569 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b4af0c27, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e7ad95b2}, log id: 2f1dc196 >2014-01-28 00:14:07,585 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-91) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:07,592 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-91) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:07,641 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterTasksListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 7e395c08 >2014-01-28 00:14:07,679 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterTasksListVDSCommand, log id: 7e395c08 >2014-01-28 00:14:07,681 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-92) org.ovirt.engine.core.common.errors.VDSError@146001f >2014-01-28 00:14:07,682 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-92) Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed >error: tasks is not a valid status option >Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool] >return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161) > at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-01-28 00:14:09,435 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-14) [6980b72e] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 Type: VM >2014-01-28 00:14:09,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-14) [6980b72e] START, SetVmTicketVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3, ticket=cJRhRjxGzEou, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 3ed25dc2 >2014-01-28 00:14:09,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-14) [6980b72e] FINISH, SetVmTicketVDSCommand, log id: 3ed25dc2 >2014-01-28 00:14:09,611 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-14) [6980b72e] Correlation ID: 6980b72e, Call Stack: null, Custom Event ID: -1, Message: user admin@internal initiated console session for VM dev1-melb-example-net >2014-01-28 00:14:12,680 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 1a018206 >2014-01-28 00:14:12,761 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:12,769 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:12,771 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8b68d31f, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8fcfdce8}, log id: 1a018206 >2014-01-28 00:14:12,775 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:12,777 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:17,863 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 6ec8bd13 >2014-01-28 00:14:17,915 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-50) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:17,917 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-50) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:17,919 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e8fb3859, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1b2f0743}, log id: 6ec8bd13 >2014-01-28 00:14:17,922 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-50) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:17,924 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-50) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:23,005 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 165ff525 >2014-01-28 00:14:28,365 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-01-28 00:14:28,367 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, GlusterVolumesListVDSCommand, log id: 165ff525 >2014-01-28 00:14:28,377 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetStatsVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-01-28 00:14:28,385 ERROR [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-11) Error while refreshing Gluster lightweight data of cluster Data_Hall_8!: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset (Failed with error VDS_NETWORK_ERROR and code 5022) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterJob.runVdsCommand(GlusterJob.java:65) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.fetchVolumes(GlusterSyncJob.java:392) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.fetchVolumes(GlusterSyncJob.java:378) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshVolumeData(GlusterSyncJob.java:349) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshClusterData(GlusterSyncJob.java:105) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:84) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-01-28 00:14:28,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [79ca67a2] Command SpmStatusVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-01-28 00:14:28,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) vds::refreshVdsStats Failed getVdsStats, vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, error = VDSNetworkException: java.net.SocketException: Connection reset >2014-01-28 00:14:28,453 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2014-01-28 00:14:28,464 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Failed to refresh VDS , vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-01-28 00:14:28,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [4ae9a898] Correlation ID: 4ae9a898, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NextDC_M1. Setting Data Center status to Non Responsive (On host HV01, Error: Network error during communication with the Host.). >2014-01-28 00:14:28,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-01-28 00:14:31,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@42aa46c6 >2014-01-28 00:14:31,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) HostName = HV01 >2014-01-28 00:14:31,581 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-01-28 00:14:31,655 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:537) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:335) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host HV01 is initializing. Message: Recovering from crash or Initializing >2014-01-28 00:14:31,682 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Failed to refresh VDS , vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, error = Recovering from crash or Initializing, continuing. >2014-01-28 00:14:34,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) [221e4d0a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78b87d34 >2014-01-28 00:14:34,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) [221e4d0a] HostName = HV01 >2014-01-28 00:14:34,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) [221e4d0a] Command GetCapabilitiesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-01-28 00:14:37,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7b19947c >2014-01-28 00:14:37,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) HostName = HV01 >2014-01-28 00:14:37,973 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-01-28 00:14:38,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-01-28 00:14:38,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] HostName = HV01 >2014-01-28 00:14:38,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] Command HSMGetAllTasksStatusesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-01-28 00:14:41,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] START, GetHardwareInfoVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01]), log id: 1c3aeb76 >2014-01-28 00:14:41,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] FINISH, GetHardwareInfoVDSCommand, log id: 1c3aeb76 >2014-01-28 00:14:41,669 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] START, GlusterServersListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 21d18ebd >2014-01-28 00:14:41,771 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] FINISH, GlusterServersListVDSCommand, return: [172.16.0.11:CONNECTED], log id: 21d18ebd >2014-01-28 00:14:41,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [430df2fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host HV01. >2014-01-28 00:14:41,837 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [430df2fc] VM dev1-melb-example-net 9c9befe0-9e17-462e-9bce-27ddd684a6d3 moved from Up --> Paused >2014-01-28 00:14:41,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] START, FullListVdsCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01], vmIds=[9c9befe0-9e17-462e-9bce-27ddd684a6d3]), log id: 65966bac >2014-01-28 00:14:41,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-24) [430df2fc] FINISH, FullListVdsCommand, return: [Ljava.util.HashMap;@76771d4e, log id: 65966bac >2014-01-28 00:14:41,868 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) [430df2fc] Cancelling the recovery from crash timer for VDS 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 because vds started initializing >2014-01-28 00:14:41,870 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-24) [1f65ffc5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2014-01-28 00:14:41,878 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2014-01-28 00:14:41,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, ConnectStorageServerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = GLUSTERFS, connectionList = [{ id: 594fb154-104f-4248-b87b-4d01cf9b5325, connection: 172.16.1.5:/VM-DATA, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cc660c4 >2014-01-28 00:14:41,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, ConnectStorageServerVDSCommand, return: {594fb154-104f-4248-b87b-4d01cf9b5325=0}, log id: 6cc660c4 >2014-01-28 00:14:41,993 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] Host HV01 storage connection was succeeded >2014-01-28 00:14:42,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, ConnectStorageServerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: f414e026-440e-46ac-ac3f-213042ee2cce, connection: engine.melb.example.net:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5200630c >2014-01-28 00:14:42,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, ConnectStorageServerVDSCommand, return: {f414e026-440e-46ac-ac3f-213042ee2cce=0}, log id: 5200630c >2014-01-28 00:14:42,067 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] Host HV01 storage connection was succeeded >2014-01-28 00:14:42,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) START, ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1), log id: 16acf4c0 >2014-01-28 00:14:42,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d']] >2014-01-28 00:14:42,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) HostName = HV01 >2014-01-28 00:14:42,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) Command ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d' >2014-01-28 00:14:42,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) FINISH, ConnectStoragePoolVDSCommand, log id: 16acf4c0 >2014-01-28 00:14:42,596 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) Could not connect host HV01 to pool NextDC_M1, as the master domain is in inactive/unknown status - not failing the operation >2014-01-28 00:14:42,821 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-24) [143a11c8] Failed to run Power Management command on Host HV01, no running proxy Host was found. >2014-01-28 00:14:42,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, SetMOMPolicyParametersVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 765621a4 >2014-01-28 00:14:42,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 765621a4 >2014-01-28 00:14:42,951 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] START, GetGlusterHostUUIDVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 10bad2ae >2014-01-28 00:14:42,998 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler_Worker-24) [143a11c8] FINISH, GetGlusterHostUUIDVDSCommand, return: 1fba63f6-2109-40a1-bd33-e051fa939b81, log id: 10bad2ae >2014-01-28 00:14:43,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [143a11c8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host HV01.There is no other host in the data center that can be used to test the power management settings. >2014-01-28 00:14:43,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [143a11c8] Correlation ID: 1f65ffc5, Call Stack: null, Custom Event ID: -1, Message: Host HV01 power management was verified successfully. >2014-01-28 00:14:43,136 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [b0595b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 Type: VDS >2014-01-28 00:14:43,168 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [7c462c25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 Type: VDS >2014-01-28 00:14:43,179 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-44) domain 436c344c-bc57-441b-b311-c9595c6039e1:ISOS in problem. vds: HV01 >2014-01-28 00:14:43,593 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 522fec1f >2014-01-28 00:14:43,656 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-9) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:43,666 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-9) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:43,674 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@28eabbd2, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@dd83881a}, log id: 522fec1f >2014-01-28 00:14:43,690 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-9) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:43,698 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-9) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:46,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-16) START, FullListVdsCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vds=Host[HV01], vmIds=[e85be056-7715-485c-9cb8-69f9980ed3a3, 48ad952b-4457-4ae1-bf85-93f869382298, 546b9733-3453-4210-90fd-2b79cf08d53b, bbbd4efa-022d-44d3-8274-a0506bda2d5e, b3cde33c-f4c5-4b42-8e9d-820b80abc2fb, 01d705e3-6b62-4796-b9e4-4de1c477401a]), log id: 74850c6b >2014-01-28 00:14:46,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FullListVdsCommand, return: [Ljava.util.HashMap;@4af28090, log id: 74850c6b >2014-01-28 00:14:46,358 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Received a bridge Device without an address when processing VM 48ad952b-4457-4ae1-bf85-93f869382298 devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:19:42:e0, linkActive=true, type=interface, deviceId=3200b542-606b-4e19-a123-9082787ca210, filter=vdsm-no-mac-spoofing, network=r2o_net} >2014-01-28 00:14:46,374 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Received a bridge Device without an address when processing VM b3cde33c-f4c5-4b42-8e9d-820b80abc2fb devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:19:42:e1, linkActive=true, type=interface, deviceId=e80486d7-7833-477b-829e-4198688db798, filter=vdsm-no-mac-spoofing, network=r2o_net} >2014-01-28 00:14:46,392 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Received a bridge Device without an address when processing VM 546b9733-3453-4210-90fd-2b79cf08d53b devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:19:42:1c, linkActive=true, type=interface, deviceId=7aa48c2d-acdf-41d4-972f-05cd92c5892a, filter=vdsm-no-mac-spoofing, network=r2o_net} >2014-01-28 00:14:48,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] hostFromVds::selectedVds - HV01, spmStatus Unknown_Pool, storage pool NextDC_M1 >2014-01-28 00:14:48,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] START, ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1), log id: 5c03afee >2014-01-28 00:14:48,791 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3ae772d6 >2014-01-28 00:14:48,867 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-22) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:48,869 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-22) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:48,871 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b22933bc, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@737c1ab}, log id: 3ae772d6 >2014-01-28 00:14:48,874 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-22) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:48,876 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-22) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:49,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d']] >2014-01-28 00:14:49,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] HostName = HV01 >2014-01-28 00:14:49,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] Command ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d' >2014-01-28 00:14:49,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] FINISH, ConnectStoragePoolVDSCommand, log id: 5c03afee >2014-01-28 00:14:49,209 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-01-28 00:14:49,214 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [66f2577c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d' >2014-01-28 00:14:49,407 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: ff3a0c94-11dc-446d-b093-5f68ad81520d Type: Storage >2014-01-28 00:14:49,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] START, ResetIrsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, vdsId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, ignoreStopFailed = true), log id: 2176ee7f >2014-01-28 00:14:49,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] START, SpmStopVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3), log id: 1a389302 >2014-01-28 00:14:49,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-01-28 00:14:49,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] HostName = HV01 >2014-01-28 00:14:49,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Command HSMGetAllTasksStatusesVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-01-28 00:14:49,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] FINISH, SpmStopVDSCommand, log id: 1a389302 >2014-01-28 00:14:49,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-20) [156ff99d] FINISH, ResetIrsVDSCommand, log id: 2176ee7f >2014-01-28 00:14:49,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [156ff99d] Correlation ID: 156ff99d, Job ID: 20100c6d-068c-4445-9d35-91a669e15dc5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center NextDC_M1. >2014-01-28 00:14:49,638 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-39) [66f2577c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2014-01-28 00:14:53,965 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 25ba21d5 >2014-01-28 00:14:54,023 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:54,027 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-97) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:54,029 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@71e5a6ba, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c6be54ea}, log id: 25ba21d5 >2014-01-28 00:14:54,034 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:54,037 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-97) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:59,120 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) [2697e1d7] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 35ef9d >2014-01-28 00:14:59,206 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:59,210 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:14:59,211 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) [2697e1d7] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@21989f42, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@eaa0a25}, log id: 35ef9d >2014-01-28 00:14:59,215 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:59,216 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-71) [2697e1d7] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:14:59,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) hostFromVds::selectedVds - HV01, spmStatus Unknown_Pool, storage pool NextDC_M1 >2014-01-28 00:14:59,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) START, ConnectStoragePoolVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = ff3a0c94-11dc-446d-b093-5f68ad81520d, masterVersion = 1), log id: 52216d1c >2014-01-28 00:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) Autorecovering 1 storage domains >2014-01-28 00:15:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) Autorecovering storage domains id: ff3a0c94-11dc-446d-b093-5f68ad81520d >2014-01-28 00:15:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-41) [5e0d5090] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: ff3a0c94-11dc-446d-b093-5f68ad81520d Type: Storage >2014-01-28 00:15:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-41) [5e0d5090] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/28/14 12:15 AM >2014-01-28 00:15:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-43) [2200647a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-01-28 00:15:00,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [2200647a] START, ConnectStorageServerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 594fb154-104f-4248-b87b-4d01cf9b5325, connection: 172.16.1.5:/VM-DATA, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50796f8b >2014-01-28 00:15:00,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [2200647a] FINISH, ConnectStorageServerVDSCommand, return: {594fb154-104f-4248-b87b-4d01cf9b5325=0}, log id: 50796f8b >2014-01-28 00:15:00,571 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-41) [5e0d5090] ConnectDomainToStorage. After Connect all hosts to pool. Time:1/28/14 12:15 AM >2014-01-28 00:15:03,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, ConnectStoragePoolVDSCommand, log id: 52216d1c >2014-01-28 00:15:03,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) hostFromVds::Connected host to pool - selectedVds - HV01, spmStatus Free, storage pool NextDC_M1 >2014-01-28 00:15:04,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) SpmStatus on vds 723cdb54-f44c-4ffd-b1bf-fde7ec606d92: Free >2014-01-28 00:15:04,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) starting spm on vds HV01, storage pool NextDC_M1, prevId 1, LVER 1 >2014-01-28 00:15:04,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) START, SpmStartVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, prevId=1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a177e76 >2014-01-28 00:15:04,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling started: taskId = 4155532d-c8c0-4274-8551-25b0193797b3 >2014-01-28 00:15:04,348 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 48f93b85 >2014-01-28 00:15:04,469 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:04,480 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-46) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:04,490 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@689bbf9, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@df26eff8}, log id: 48f93b85 >2014-01-28 00:15:04,510 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:04,519 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-46) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:07,760 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) START, GlusterTasksListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 39bf287a >2014-01-28 00:15:07,844 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, GlusterTasksListVDSCommand, log id: 39bf287a >2014-01-28 00:15:07,850 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-56) org.ovirt.engine.core.common.errors.VDSError@24c8951f >2014-01-28 00:15:07,856 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-56) Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed >error: tasks is not a valid status option >Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool] >return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161) > at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-01-28 00:15:09,606 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 4e28d6be >2014-01-28 00:15:09,656 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-13) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:09,666 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-13) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:09,674 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fc1070e2, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1e780c70}, log id: 4e28d6be >2014-01-28 00:15:09,689 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-13) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:09,696 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-13) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:14,788 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3c6fdaf6 >2014-01-28 00:15:14,875 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:14,884 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:14,893 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fd92fde5, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7f2f8396}, log id: 3c6fdaf6 >2014-01-28 00:15:14,909 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-61) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:14,917 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-61) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:20,011 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) [44f165b7] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3b1eb02c >2014-01-28 00:15:20,076 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-69) [44f165b7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:20,086 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-69) [44f165b7] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:20,095 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) [44f165b7] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@81f1741c, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@dd5cefb4}, log id: 3b1eb02c >2014-01-28 00:15:20,111 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-69) [44f165b7] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:20,118 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-69) [44f165b7] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:22,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling ended: taskId = 4155532d-c8c0-4274-8551-25b0193797b3 task status = finished >2014-01-28 00:15:22,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) spmStart polling ended, spm status: SPM >2014-01-28 00:15:22,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-91) START, HSMClearTaskVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, taskId=4155532d-c8c0-4274-8551-25b0193797b3), log id: 18beda7d >2014-01-28 00:15:22,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, HSMClearTaskVDSCommand, log id: 18beda7d >2014-01-28 00:15:22,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71d4b4c7, log id: 5a177e76 >2014-01-28 00:15:22,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) Initialize Irs proxy from vds: 172.16.0.11 >2014-01-28 00:15:22,800 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host HV01 (Address: 172.16.0.11). >2014-01-28 00:15:25,210 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [6be25d2] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: fe55cf7 >2014-01-28 00:15:25,269 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-59) [6be25d2] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:25,273 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-59) [6be25d2] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:25,276 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [6be25d2] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e97bf33a, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ef3e059e}, log id: fe55cf7 >2014-01-28 00:15:25,281 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-59) [6be25d2] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:25,283 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-59) [6be25d2] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:30,365 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 18f6c356 >2014-01-28 00:15:30,456 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-5) [195dc414] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:30,466 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-5) [195dc414] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:30,475 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-5) [195dc414] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a3d5ce6d, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@57b0a1e9}, log id: 18f6c356 >2014-01-28 00:15:30,492 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-5) [195dc414] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:30,500 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-5) [195dc414] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:35,151 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) Storage Domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA was reported by Host HV01 as Active in Pool 5849b030-626e-47cb-ad90-3ce782d831b3, moving to active status >2014-01-28 00:15:35,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA recovered from problem. vds: HV01 >2014-01-28 00:15:35,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain ff3a0c94-11dc-446d-b093-5f68ad81520d:VM-DATA has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-01-28 00:15:35,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain 436c344c-bc57-441b-b311-c9595c6039e1:ISOS recovered from problem. vds: HV01 >2014-01-28 00:15:35,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-12) Domain 436c344c-bc57-441b-b311-c9595c6039e1:ISOS has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-01-28 00:15:35,592 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 60c6e137 >2014-01-28 00:15:35,644 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-96) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:35,646 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-96) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:35,648 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@126e824a, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a9d97791}, log id: 60c6e137 >2014-01-28 00:15:35,651 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-96) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:35,652 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-96) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:40,743 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) [4c514a31] START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 698797b8 >2014-01-28 00:15:40,816 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [4c514a31] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:40,825 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [4c514a31] Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:40,833 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) [4c514a31] FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9f4386aa, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2e15b648}, log id: 698797b8 >2014-01-28 00:15:40,848 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-53) [4c514a31] Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:40,855 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-53) [4c514a31] Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:45,947 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 5c40bd0b >2014-01-28 00:15:46,042 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-84) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:46,045 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-84) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:46,047 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b2bad5a7, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@114ae154}, log id: 5c40bd0b >2014-01-28 00:15:46,051 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-84) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:46,053 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-84) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:51,134 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 647a7e41 >2014-01-28 00:15:51,186 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:51,189 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-79) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:51,190 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e9746c6d, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4adeb311}, log id: 647a7e41 >2014-01-28 00:15:51,193 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:51,194 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-79) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:56,278 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 4c492531 >2014-01-28 00:15:56,330 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-72) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:56,332 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-72) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:15:56,334 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@572dc7be, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@98a37c44}, log id: 4c492531 >2014-01-28 00:15:56,337 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-72) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:56,340 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-72) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:15:58,015 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-3) [6aad9c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 value: VM >, sharedLocks= ] >2014-01-28 00:15:58,090 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Running command: RunVmCommand internal: false. Entities affected : ID: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 Type: VM >2014-01-28 00:15:58,169 INFO [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] START, ResumeVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3), log id: 5c1bea93 >2014-01-28 00:15:58,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] START, ResumeBrokerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3), log id: 57a2216 >2014-01-28 00:15:58,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Failed in ResumeBrokerVDS method >2014-01-28 00:15:58,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.OneVmReturnForXmlRpc@5d0d34f7 >2014-01-28 00:15:58,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] HostName = HV01 >2014-01-28 00:15:58,239 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Command ResumeBrokerVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92, vmId=9c9befe0-9e17-462e-9bce-27ddd684a6d3) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to ResumeBrokerVDS, error = Unexpected exception, code = 16 >2014-01-28 00:15:58,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] FINISH, ResumeBrokerVDSCommand, log id: 57a2216 >2014-01-28 00:15:58,242 ERROR [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] VDS::pause Failed resume vm 9c9befe0-9e17-462e-9bce-27ddd684a6d3 in vds = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92 : HV01, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ResumeBrokerVDS, error = Unexpected exception, code = 16 >2014-01-28 00:15:58,244 INFO [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] FINISH, ResumeVDSCommand, return: Unknown, log id: 5c1bea93 >2014-01-28 00:15:58,246 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Lock freed to object EngineLock [exclusiveLocks= key: 9c9befe0-9e17-462e-9bce-27ddd684a6d3 value: VM >, sharedLocks= ] >2014-01-28 00:15:58,254 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Command org.ovirt.engine.core.bll.RunVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ResumeBrokerVDS, error = Unexpected exception, code = 16 (Failed with error unexpected and code 16) >2014-01-28 00:15:58,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [6aad9c7] Correlation ID: 6aad9c7, Job ID: e58fee76-8475-4492-a7f3-74c77e3956cc, Call Stack: null, Custom Event ID: -1, Message: Failed to resume VM dev1-melb-example-net (Host: HV01, User: admin@internal). >2014-01-28 00:16:01,430 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-67) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 159c155d >2014-01-28 00:16:01,497 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-67) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:16:01,500 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-67) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:16:01,502 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a560d359, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe3b4ddd}, log id: 159c155d >2014-01-28 00:16:01,508 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-67) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:16:01,510 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-67) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:16:06,594 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 4cd27c70 >2014-01-28 00:16:06,644 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-19) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:16:06,646 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-19) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:16:06,647 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c768bcd7, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@591f9b20}, log id: 4cd27c70 >2014-01-28 00:16:06,651 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-19) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:16:06,652 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-19) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:16:07,964 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] START, GlusterTasksListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: a023379 >2014-01-28 00:16:08,007 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-6) [4ae9a898] FINISH, GlusterTasksListVDSCommand, log id: a023379 >2014-01-28 00:16:08,009 ERROR [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler_Worker-6) [4ae9a898] org.ovirt.engine.core.common.errors.VDSError@74067b75 >2014-01-28 00:16:08,010 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler_Worker-6) [4ae9a898] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Command execution failed >error: tasks is not a valid status option >Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool] >return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161) > at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:43) [bll.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:84) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-01-28 00:16:11,734 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GlusterVolumesListVDSCommand(HostName = HV01, HostId = 723cdb54-f44c-4ffd-b1bf-fde7ec606d92), log id: 3767e8ac >2014-01-28 00:16:11,800 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:16:11,806 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-70) Could not find server gs01.melb.example.net in cluster 99408929-82cf-4dc7-a532-9d998063fa95 >2014-01-28 00:16:11,808 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GlusterVolumesListVDSCommand, return: {f8e51af1-a0ca-411b-9738-c3e673d6c4e3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f5fe2e7b, 224dc5cc-3ce9-4db2-84dc-4694c8bd6759=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b70ce26a}, log id: 3767e8ac >2014-01-28 00:16:11,818 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume VM-DATA contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >2014-01-28 00:16:11,826 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-70) Volume HOSTED-ENGINE contains brick(s) from unknown hosts. Hence will not add it to engine at this point. >^C
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1058300
:
856063
|
856064
|
858008
|
858009
|
858010
|
867646
|
867647
|
884735
|
909506