Attachment 860167 Details for Bug 1062228 – engine.log

engine.log

engine.log (text/x-log), 46.99 KB, created by Artyom on 2014-02-06 13:34:22 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Artyom

Created: 2014-02-06 13:34:22 UTC

Size: 46.99 KB

patch

obsolete

>2014-02-06 14:49:55,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-5) [582e5430] START, DestroyVDSCommand(HostName = cyan-vdsf.qa.lab.tlv.redhat.com, HostId = 9ec23676-348b-4586-b516-de773eba82a9, vmId=99fb2a6b-93dc-4cb0-a199-237cf96466c4, force=false, secondsToWait=0, gracefully=false), log id: 41db65d6
>2014-02-06 14:49:55,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-5) [582e5430] FINISH, DestroyVDSCommand, log id: 41db65d6
>2014-02-06 14:49:55,053 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [582e5430] RefreshVmList vm id 99fb2a6b-93dc-4cb0-a199-237cf96466c4 status = Down on vds cyan-vdsf.qa.lab.tlv.redhat.com ignoring it in the refresh until migration is done
>2014-02-06 14:50:50,356 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-69) Automatic power management is disabled for cluster cl_34.
>2014-02-06 14:51:50,486 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-45) Automatic power management is disabled for cluster cl_34.
>2014-02-06 14:52:50,616 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-33) Automatic power management is disabled for cluster cl_34.
>2014-02-06 14:53:08,333 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp--127.0.0.1-8702-3) [4174373] Running command: UpdateVdsGroupCommand internal: false. Entities affected :  ID: 671205b6-e349-4346-af45-d7bb8dd982b0 Type: VdsGroups
>2014-02-06 14:53:08,346 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4174373] Correlation ID: 4174373, Call Stack: null, Custom Event ID: -1, Message: Host cluster cl_34 was updated by admin
>2014-02-06 14:53:50,749 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com is not currently needed, activating maintenance mode in preparation for shutdown.
>2014-02-06 14:53:50,876 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (DefaultQuartzScheduler_Worker-35) [1a15cee6] Running command: MaintenanceNumberOfVdssCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 14:53:50,923 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [1a15cee6] START, SetVdsStatusVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 5f1f0ffe
>2014-02-06 14:53:50,976 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [1a15cee6] FINISH, SetVdsStatusVDSCommand, log id: 5f1f0ffe
>2014-02-06 14:53:51,103 INFO  [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (DefaultQuartzScheduler_Worker-35) [1a15cee6] Running command: MaintenanceVdsCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 14:53:51,113 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [1a15cee6] Correlation ID: 1a15cee6, Job ID: 6299e790-063d-491f-bbf0-90e79ed3b67e, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com was switched to Maintenance Mode.
>2014-02-06 14:53:54,031 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 : rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:53:54,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-8) Clearing cache of pool: fba8c24a-e0f0-4813-8e21-984403b3a62b for problematic entities of VDS: rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:53:54,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) START, DisconnectStoragePoolVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, storagePoolId = fba8c24a-e0f0-4813-8e21-984403b3a62b, vds_spm_id = 2), log id: 1a4c7657
>2014-02-06 14:53:56,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, DisconnectStoragePoolVDSCommand, log id: 1a4c7657
>2014-02-06 14:53:56,091 INFO  [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [295fad9e] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected :  ID: fba8c24a-e0f0-4813-8e21-984403b3a62b Type: StoragePool
>2014-02-06 14:53:56,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [295fad9e] START, DisconnectStorageServerVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, storagePoolId = fba8c24a-e0f0-4813-8e21-984403b3a62b, storageType = NFS, connectionList = [{ id: aa540447-c67e-4611-ad50-e0fe7e47ed6b, connection: 10.35.160.108:/RHEV/artyom_ovirt34, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1510695c
>2014-02-06 14:53:56,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [295fad9e] FINISH, DisconnectStorageServerVDSCommand, return: {aa540447-c67e-4611-ad50-e0fe7e47ed6b=0}, log id: 1510695c
>2014-02-06 14:54:51,337 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-42) Cluster cl_34 does have enough spare hosts, shutting one host down.
>2014-02-06 14:54:51,346 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com is not currently needed, shutting down.
>2014-02-06 14:54:51,359 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (DefaultQuartzScheduler_Worker-42) [3cb4a1ef] The message key VdsPowerDown is missing from bundles/ExecutionMessages
>2014-02-06 14:54:51,405 INFO  [org.ovirt.engine.core.bll.VdsPowerDownCommand] (DefaultQuartzScheduler_Worker-42) [3cb4a1ef] Running command: VdsPowerDownCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 14:54:51,439 INFO  [org.ovirt.engine.core.bll.VdsPowerDownCommand] (DefaultQuartzScheduler_Worker-42) [3cb4a1ef] Opening SSH power down session on host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:54:51,918 INFO  [org.ovirt.engine.core.bll.VdsPowerDownCommand] (DefaultQuartzScheduler_Worker-42) [3cb4a1ef] Executing SSH power down command on host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:55:52,159 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-18) Cluster cl_34 does have enough spare hosts, shutting one host down.
>2014-02-06 14:55:52,163 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com is not currently needed, shutting down.
>2014-02-06 14:55:52,170 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (DefaultQuartzScheduler_Worker-18) [7a71bcec] The message key VdsPowerDown is missing from bundles/ExecutionMessages
>2014-02-06 14:55:52,213 INFO  [org.ovirt.engine.core.bll.VdsPowerDownCommand] (DefaultQuartzScheduler_Worker-18) [7a71bcec] Running command: VdsPowerDownCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 14:55:52,248 INFO  [org.ovirt.engine.core.bll.VdsPowerDownCommand] (DefaultQuartzScheduler_Worker-18) [7a71bcec] Opening SSH power down session on host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:56:52,328 ERROR [org.ovirt.engine.core.bll.VdsPowerDownCommand] (DefaultQuartzScheduler_Worker-18) [7a71bcec] SSH power down command failed on host rose05.qa.lab.tlv.redhat.com: Failed to get the session.
>Stdout: 
>Stderr: 
>Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session.
>    at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43)
>    at org.ovirt.engine.core.utils.ssh.SSHClient.connect(SSHClient.java:294) [utils.jar:]
>    at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:]
>    at org.ovirt.engine.core.bll.VdsPowerDownCommand.executeSshPowerdown(VdsPowerDownCommand.java:95) [bll.jar:]
>    at org.ovirt.engine.core.bll.VdsPowerDownCommand.executeCommand(VdsPowerDownCommand.java:47) [bll.jar:]
>    at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1114) [bll.jar:]
>    at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1199) [bll.jar:]
>    at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1875) [bll.jar:]
>    at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
>    at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
>    at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1219) [bll.jar:]
>    at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:]
>    at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:413) [bll.jar:]
>    at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:392) [bll.jar:]
>    at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:624) [bll.jar:]
>    at sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) [:1.7.0_51]
>    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>    at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>    at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>    at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:]
>    at org.ovirt.engine.core.bll.scheduling.policyunits.PowerSavingBalancePolicyUnit.processPmAction(PowerSavingBalancePolicyUnit.java:98) [bll.jar:]
>    at org.ovirt.engine.core.bll.scheduling.policyunits.PowerSavingBalancePolicyUnit.balance(PowerSavingBalancePolicyUnit.java:64) [bll.jar:]
>    at org.ovirt.engine.core.bll.scheduling.SchedulingManager.internalRunBalance(SchedulingManager.java:822) [bll.jar:]
>    at org.ovirt.engine.core.bll.scheduling.SchedulingManager.performLoadBalancing(SchedulingManager.java:809) [bll.jar:]
>    at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) [:1.7.0_51]
>    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
>    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
>    at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>    at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection timed out.
>    at org.apache.mina.core.polling.AbstractPollingIoConnector.processTimedOutSessions(AbstractPollingIoConnector.java:455) [mina-core.jar:]
>    at org.apache.mina.core.polling.AbstractPollingIoConnector.access$600(AbstractPollingIoConnector.java:62) [mina-core.jar:]
>    at org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:479) [mina-core.jar:]
>    at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [mina-core.jar:]
>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
>    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
>
>2014-02-06 14:56:52,458 INFO  [org.ovirt.engine.core.bll.StopVdsCommand] (DefaultQuartzScheduler_Worker-18) [4dbc620a] Lock Acquired to object EngineLock [exclusiveLocks= key: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 value: VDS_FENCE
>, sharedLocks= ]
>2014-02-06 14:56:52,708 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:56:52,710 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:56:52,715 INFO  [org.ovirt.engine.core.bll.StopVdsCommand] (DefaultQuartzScheduler_Worker-18) Running command: StopVdsCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 14:56:52,753 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) START, SetVdsStatusVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, status=Down, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: fde0b4d
>2014-02-06 14:56:52,758 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, SetVdsStatusVDSCommand, log id: fde0b4d
>2014-02-06 14:56:52,886 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Stop command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:56:52,888 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Stop command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:56:52,919 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Stop> Power Management command, Proxy Host:brown-vdsc.qa.lab.tlv.redhat.com, Agent:ipmilan, Target Host:rose05.qa.lab.tlv.redhat.com, Management IP:rose05-mgmt.qa.lab.tlv.redhat.com, User:root, Options:
>2014-02-06 14:56:52,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = brown-vdsc.qa.lab.tlv.redhat.com, HostId = cf6a768d-b47d-4781-a7b1-425d2ec9c784, targetVdsId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, action = Stop, ip = rose05-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 83716f0
>2014-02-06 14:56:53,051 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com is already off, Power Management Stop operation skipped.
>2014-02-06 14:56:53,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Succeeded, skipped, log id: 83716f0
>2014-02-06 14:56:53,083 INFO  [org.ovirt.engine.core.bll.StopVdsCommand] (DefaultQuartzScheduler_Worker-18) Waiting for vds rose05.qa.lab.tlv.redhat.com to stop
>2014-02-06 14:56:58,086 INFO  [org.ovirt.engine.core.bll.StopVdsCommand] (DefaultQuartzScheduler_Worker-18) Attempt 1 to get vds rose05.qa.lab.tlv.redhat.com status
>2014-02-06 14:56:58,189 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:56:58,191 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:56:58,224 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:brown-vdsc.qa.lab.tlv.redhat.com, Agent:ipmilan, Target Host:rose05.qa.lab.tlv.redhat.com, Management IP:rose05-mgmt.qa.lab.tlv.redhat.com, User:root, Options:
>2014-02-06 14:56:58,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = brown-vdsc.qa.lab.tlv.redhat.com, HostId = cf6a768d-b47d-4781-a7b1-425d2ec9c784, targetVdsId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, action = Status, ip = rose05-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1560fa3c
>2014-02-06 14:56:58,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 1560fa3c
>2014-02-06 14:56:58,394 INFO  [org.ovirt.engine.core.bll.StopVdsCommand] (DefaultQuartzScheduler_Worker-18) vds rose05.qa.lab.tlv.redhat.com status is off
>2014-02-06 14:56:58,400 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: 4dbc620a, Job ID: b532a764-538d-4ac0-a54a-2022441134ac, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com was stopped by Engine.
>2014-02-06 14:56:58,404 INFO  [org.ovirt.engine.core.bll.StopVdsCommand] (DefaultQuartzScheduler_Worker-18) Lock freed to object EngineLock [exclusiveLocks= key: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 value: VDS_FENCE
>, sharedLocks= ]
>2014-02-06 14:57:58,743 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp--127.0.0.1-8702-6) [159e12ff] Running command: UpdateVdsGroupCommand internal: false. Entities affected :  ID: 671205b6-e349-4346-af45-d7bb8dd982b0 Type: VdsGroups
>2014-02-06 14:57:58,751 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [159e12ff] Correlation ID: 159e12ff, Call Stack: null, Custom Event ID: -1, Message: Host cluster cl_34 was updated by admin
>2014-02-06 14:58:58,616 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-61) [7bd28672] Cluster cl_34 does not have enough spare hosts, trying to start one up.
>2014-02-06 14:58:58,628 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [7bd28672] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Reactivating host rose05.qa.lab.tlv.redhat.com according to the current power management policy.
>2014-02-06 14:58:58,675 INFO  [org.ovirt.engine.core.bll.StartVdsCommand] (DefaultQuartzScheduler_Worker-61) [2b6dc9c] Lock Acquired to object EngineLock [exclusiveLocks= key: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 value: VDS_FENCE
>, sharedLocks= ]
>2014-02-06 14:58:58,808 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:58:58,810 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:58:58,816 INFO  [org.ovirt.engine.core.bll.StartVdsCommand] (DefaultQuartzScheduler_Worker-61) Running command: StartVdsCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 14:58:58,850 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) START, SetVdsStatusVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6429ba5b
>2014-02-06 14:58:58,855 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, SetVdsStatusVDSCommand, log id: 6429ba5b
>2014-02-06 14:58:58,985 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Start command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:58:58,987 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Start command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:58:59,020 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Start> Power Management command, Proxy Host:brown-vdsc.qa.lab.tlv.redhat.com, Agent:ipmilan, Target Host:rose05.qa.lab.tlv.redhat.com, Management IP:rose05-mgmt.qa.lab.tlv.redhat.com, User:root, Options:
>2014-02-06 14:58:59,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = brown-vdsc.qa.lab.tlv.redhat.com, HostId = cf6a768d-b47d-4781-a7b1-425d2ec9c784, targetVdsId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, action = Start, ip = rose05-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 624c33c6
>2014-02-06 14:58:59,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Done, log id: 624c33c6
>2014-02-06 14:58:59,137 INFO  [org.ovirt.engine.core.bll.StartVdsCommand] (DefaultQuartzScheduler_Worker-61) Waiting for vds rose05.qa.lab.tlv.redhat.com to start
>2014-02-06 14:59:01,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:04,139 INFO  [org.ovirt.engine.core.bll.StartVdsCommand] (DefaultQuartzScheduler_Worker-61) Attempt 1 to get vds rose05.qa.lab.tlv.redhat.com status
>2014-02-06 14:59:04,238 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 14:59:04,241 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 14:59:04,277 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:brown-vdsc.qa.lab.tlv.redhat.com, Agent:ipmilan, Target Host:rose05.qa.lab.tlv.redhat.com, Management IP:rose05-mgmt.qa.lab.tlv.redhat.com, User:root, Options:
>2014-02-06 14:59:04,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = brown-vdsc.qa.lab.tlv.redhat.com, HostId = cf6a768d-b47d-4781-a7b1-425d2ec9c784, targetVdsId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, action = Status, ip = rose05-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 23c3da9a
>2014-02-06 14:59:04,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 23c3da9a
>2014-02-06 14:59:04,417 INFO  [org.ovirt.engine.core.bll.StartVdsCommand] (DefaultQuartzScheduler_Worker-61) vds rose05.qa.lab.tlv.redhat.com status is on
>2014-02-06 14:59:04,422 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: 2b6dc9c, Job ID: db9c4b14-4567-4272-8615-291bc9ca9cc4, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com was started by Engine.
>2014-02-06 14:59:04,428 INFO  [org.ovirt.engine.core.bll.StartVdsCommand] (DefaultQuartzScheduler_Worker-61) Lock freed to object EngineLock [exclusiveLocks= key: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 value: VDS_FENCE
>, sharedLocks= ]
>2014-02-06 14:59:06,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:11,677 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:16,725 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:21,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:26,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:31,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:36,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:41,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:47,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:52,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 14:59:57,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:02,332 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:04,526 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-63) Cluster cl_34 does not have enough spare hosts, but no additional host is available.
>2014-02-06 15:00:07,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:12,472 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:17,526 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) [3cb4a1ef] Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:22,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:27,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:32,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [19e07c5d] Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:37,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:42,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:47,810 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:52,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:00:57,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) [3cb4a1ef] Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
>2014-02-06 15:01:00,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-02-06 15:01:04,000 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-02-06 15:01:04,624 INFO  [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-79) Cluster cl_34 does not have enough spare hosts, but no additional host is available.
>2014-02-06 15:01:07,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-02-06 15:01:10,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-02-06 15:01:13,135 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-02-06 15:01:16,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GetHardwareInfoVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, vds=Host[rose05.qa.lab.tlv.redhat.com]), log id: 44f48da4
>2014-02-06 15:01:16,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GetHardwareInfoVDSCommand, log id: 44f48da4
>2014-02-06 15:01:16,874 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 15:01:16,881 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [7b6bff0b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: fba8c24a-e0f0-4813-8e21-984403b3a62b Type: StoragePool
>2014-02-06 15:01:16,887 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: fba8c24a-e0f0-4813-8e21-984403b3a62b Type: StoragePool
>2014-02-06 15:01:16,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] START, ConnectStorageServerVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, storagePoolId = fba8c24a-e0f0-4813-8e21-984403b3a62b, storageType = NFS, connectionList = [{ id: aa540447-c67e-4611-ad50-e0fe7e47ed6b, connection: 10.35.160.108:/RHEV/artyom_ovirt34, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3e4aff25
>2014-02-06 15:01:17,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] FINISH, ConnectStorageServerVDSCommand, return: {aa540447-c67e-4611-ad50-e0fe7e47ed6b=0}, log id: 3e4aff25
>2014-02-06 15:01:17,270 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] Host rose05.qa.lab.tlv.redhat.com storage connection was succeeded 
>2014-02-06 15:01:17,305 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) START, ConnectStoragePoolVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, storagePoolId = fba8c24a-e0f0-4813-8e21-984403b3a62b, vds_spm_id = 2, masterDomainId = 640a205b-ed14-40e2-ba0d-96a8e25ff645, masterVersion = 1), log id: 43d2772c
>2014-02-06 15:01:17,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) FINISH, ConnectStoragePoolVDSCommand, log id: 43d2772c
>2014-02-06 15:01:17,798 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [42e06ff0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 was chosen as a proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com.
>2014-02-06 15:01:17,799 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) [42e06ff0] Using Host brown-vdsc.qa.lab.tlv.redhat.com from cluster cl_34 as proxy to execute Status command on Host rose05.qa.lab.tlv.redhat.com
>2014-02-06 15:01:17,832 INFO  [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) [42e06ff0] Executing <Status> Power Management command, Proxy Host:brown-vdsc.qa.lab.tlv.redhat.com, Agent:ipmilan, Target Host:rose05.qa.lab.tlv.redhat.com, Management IP:rose05-mgmt.qa.lab.tlv.redhat.com, User:root, Options:
>2014-02-06 15:01:17,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] START, FenceVdsVDSCommand(HostName = brown-vdsc.qa.lab.tlv.redhat.com, HostId = cf6a768d-b47d-4781-a7b1-425d2ec9c784, targetVdsId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1, action = Status, ip = rose05-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 9814ba2
>2014-02-06 15:01:17,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 9814ba2
>2014-02-06 15:01:18,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] START, SetMOMPolicyParametersVDSCommand(HostName = rose05.qa.lab.tlv.redhat.com, HostId = 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1), log id: 41f3be66
>2014-02-06 15:01:18,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-93) [42e06ff0] FINISH, SetMOMPolicyParametersVDSCommand, log id: 41f3be66
>2014-02-06 15:01:18,059 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [42e06ff0] Correlation ID: 7b6bff0b, Call Stack: null, Custom Event ID: -1, Message: Host rose05.qa.lab.tlv.redhat.com power management was verified successfully.
>2014-02-06 15:01:18,097 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [35e61229] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 15:01:18,134 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [32210fce] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 58bfd634-1ba4-4cc1-ba0b-ab0e84ed01a1 Type: VDS
>2014-02-06 15:08:49,997 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp--127.0.0.1-8702-6) [162c5320] Running command: UpdateVdsGroupCommand internal: false. Entities affected :  ID: 671205b6-e349-4346-af45-d7bb8dd982b0 Type: VdsGroups
>2014-02-06 15:08:50,010 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [162c5320] Correlation ID: 162c5320, Call Stack: null, Custom Event ID: -1, Message: Host cluster cl_34 was updated by admin
>2014-02-06 15:28:43,673 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-23) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.

Actions: View

Attachments on bug 1062228: 860167 | 1057039 | 1057040