Created attachment 564174 [details]
engine logs.

Description of problem:


overview:
---------
noticed system doesn't behave the same, where in one data-center, status was activate, host was SPM, but storage domain were 'locked', while in fact, no commands was sent to vdsm, so using 'jstack', i dumped a stack trace to a file and noticed the following: 

Found one Java-level deadlock:
=============================
"QuartzScheduler_Worker-98":
  waiting to lock monitor 0x00007f448400df30 (object 0x00000000e3ba3b50, a java.lang.Object),
  which is held by "QuartzScheduler_Worker-38"
"QuartzScheduler_Worker-38":
  waiting to lock monitor 0x00000000025346d8 (object 0x00000000e3b973c0, a java.lang.Object),
  which is held by "QuartzScheduler_Worker-47"
"QuartzScheduler_Worker-47":
  waiting to lock monitor 0x00007f448400df30 (object 0x00000000e3ba3b50, a java.lang.Object),
  which is held by "QuartzScheduler_Worker-38"

        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
        at org.ovirt.engine.core.common.businessentities.IVdsEventListener$$$view7.MasterDomainNotOperational(Unknown Source)
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.ExecuteVDSCommand(IrsBrokerCommand.java:1607)
        - locked <0x00000000e3b973c0> (a java.lang.Object)
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.ExecuteCommand(VDSCommandBase.java:60)
        at org.ovirt.engine.core.dal.VdcCommandBase.Execute(VdcCommandBase.java:41)
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:414)
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData.ProceedStoragePoolStats(IrsBrokerCommand.java:312)
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData.access$200(IrsBrokerCommand.java:132)
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData$1.runInTransaction(IrsBrokerCommand.java:213)
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168)
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107)
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData._updatingTimer_Elapsed(IrsBrokerCommand.java:203)
        - locked <0x00000000e3b973c0> (a java.lang.Object)
        at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)

Found 1 deadlock.