Bug 843099 - [engine-core] unhandled exception during reconstruct: Cannot perform logging: java.util.UnknownFormatConversionException: Co nversion = 'i'
Summary: [engine-core] unhandled exception during reconstruct: Cannot perform logging:...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.1.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.1.0
Assignee: Maor
QA Contact: Dafna Ron
URL:
Whiteboard: storage
: 844315 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-07-25 15:16 UTC by Haim
Modified: 2016-02-10 17:09 UTC (History)
13 users (show)

Fixed In Version: SI15
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
oVirt Team: Storage
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
engine.log (23.11 KB, application/x-gzip)
2012-07-29 10:30 UTC, Haim 		no flags Details
View All

Description Haim 2012-07-25 15:16:41 UTC 
Description of problem:

the following nasty exception is not caught and handled upon failure to reconstruct master due to sanlock:

Cause:

2012-07-25 21:01:20,633 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [76bab1fd] Weird return value: Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.Statu
sForXmlRpc
mCode                         661
mMessage                      Cannot acquire host id: ('23fc84ef-2a31-4a20-a8c2-5802fea9e047', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))

2012-07-25 21:01:20,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [76bab1fd] Failed in ReconstructMasterVDS method
2012-07-25 21:01:20,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [76bab1fd] Error code unexpected and error message VDSGenericException: VDSErrorException:
 Failed to ReconstructMasterVDS, error = Cannot acquire host id: ('23fc84ef-2a31-4a20-a8c2-5802fea9e047', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))
2012-07-25 21:01:20,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [76bab1fd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand r
eturn value 


Full exception:

b35697-99af-45b3-8a28-29ad4805f8c3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55b7484f
2012-07-25 21:01:20,634 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [76bab1fd] clearing cache for problematic entities in pool ffb35697-99af-45b3-8a28-29ad4805f8c3
2012-07-25 21:01:20,634 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [76bab1fd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55b7484f
2012-07-25 21:01:20,634 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [76bab1fd] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot acquire host id: ('23fc84ef-2a31-4a20-a8c2-5802fea9e047', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))
2012-07-25 21:01:20,638 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [76bab1fd] Command [id=7524e4f3-5d29-4dda-9011-54b3f8d99dff]: Compensating CHANGED_ENTITY of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: id=548542e7-c31f-4b87-802d-583ffda2e020.
2012-07-25 21:01:20,646 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [76bab1fd] Command [id=7524e4f3-5d29-4dda-9011-54b3f8d99dff]: Compensating CHANGED_ENTITY of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: id=23fc84ef-2a31-4a20-a8c2-5802fea9e047.
2012-07-25 21:01:21,696 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [2c08f9c3] domain 23fc84ef-2a31-4a20-a8c2-5802fea9e047 in problem. vds: nott-vds2.qa.lab.tlv.redhat.com
2012-07-25 21:01:21,697 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [2c08f9c3] domain 77f493a8-078f-4e06-9506-4bd2ca4a8960 in problem. vds: nott-vds2.qa.lab.tlv.redhat.com
2012-07-25 21:01:30,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [3e219312] hostFromVds::selectedVds - nott-vds2.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool nfsPool
2012-07-25 21:01:30,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-7) [3e219312] START, ConnectStoragePoolVDSCommand(vdsId = 2fc87f7c-d404-11e1-bd4b-001a4a16970e, storagePoolId = ffb35697-99af-45b3-8a28-29ad4805f8c3, vds_spm_id = 1, masterDomainId = 548542e7-c31f-4b87-802d-583ffda2e020, masterVersion = 26), log id: cc5b09a
2012-07-25 21:01:32,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [3e219312] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value 
 Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         304
mMessage                      Cannot find master domain: 'spUUID=ffb35697-99af-45b3-8a28-29ad4805f8c3, msdUUID=548542e7-c31f-4b87-802d-583ffda2e020'


2012-07-25 21:01:32,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [3e219312] Vds: nott-vds2.qa.lab.tlv.redhat.com
2012-07-25 21:01:32,634 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-7) [3e219312] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ffb35697-99af-45b3-8a28-29ad4805f8c3, msdUUID=548542e7-c31f-4b87-802d-583ffda2e020'
2012-07-25 21:01:32,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-7) [3e219312] FINISH, ConnectStoragePoolVDSCommand, log id: cc5b09a
2012-07-25 21:01:32,635 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [3e219312] IrsBroker::Failed::GetStoragePoolInfoVDS
2012-07-25 21:01:32,635 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [3e219312] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ffb35697-99af-45b3-8a28-29ad4805f8c3, msdUUID=548542e7-c31f-4b87-802d-583ffda2e020'
2012-07-25 21:01:32,640 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-7) [cfa2e4c] The message key ReconstructMasterDomain is missing from bundles/ExecutionMessages
2012-07-25 21:01:32,897 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [cfa2e4c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 548542e7-c31f-4b87-802d-583ffda2e020 Type: Storage
2012-07-25 21:01:33,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [cfa2e4c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = ffb35697-99af-45b3-8a28-29ad4805f8c3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearJobs), log id: 12275918
2012-07-25 21:01:33,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [cfa2e4c] clear domain error-timers for pool ffb35697-99af-45b3-8a28-29ad4805f8c3
2012-07-25 21:01:33,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [cfa2e4c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12275918
2012-07-25 21:01:33,009 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (QuartzScheduler_Worker-7) [cfa2e4c] Cannot perform logging: java.util.UnknownFormatConversionException: Conversion = 'i'
        at java.util.Formatter$FormatSpecifier.conversion(Formatter.java:2646) [rt.jar:1.7.0_05-icedtea]
        at java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2675) [rt.jar:1.7.0_05-icedtea]
        at java.util.Formatter.parse(Formatter.java:2528) [rt.jar:1.7.0_05-icedtea]
        at java.util.Formatter.format(Formatter.java:2469) [rt.jar:1.7.0_05-icedtea]
        at java.util.Formatter.format(Formatter.java:2423) [rt.jar:1.7.0_05-icedtea]
        at java.lang.String.format(String.java:2845) [rt.jar:1.7.0_05-icedtea]
        at org.ovirt.engine.core.common.vdscommands.ReconstructMasterVDSCommandParameters.toString(ReconstructMasterVDSCommandParameters.java:86) [engine-common.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.toString(VDSCommandBase.java:49) [engine-vdsbroker.jar:]
        at java.text.MessageFormat.subformat(MessageFormat.java:1271) [rt.jar:1.7.0_05-icedtea]
        at java.text.MessageFormat.format(MessageFormat.java:860) [rt.jar:1.7.0_05-icedtea]
        at java.text.Format.format(Format.java:157) [rt.jar:1.7.0_05-icedtea]
        at java.text.MessageFormat.format(MessageFormat.java:836) [rt.jar:1.7.0_05-icedtea]
        at org.ovirt.engine.core.utils.log.Log.transform(Log.java:159) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.log.Log.infoFormat(Log.java:120) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.log.LoggedUtils.log(LoggedUtils.java:162) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.log.LoggedUtils.logEntry(LoggedUtils.java:80) [engine-utils.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.Execute(VdcCommandBase.java:34) [engine-dal.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:390) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:35) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1302) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand.reconstructMaster(ReconstructMasterDomainCommand.java:132) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand.executeCommand(ReconstructMasterDomainCommand.java:143) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.ExecuteWithoutTransaction(CommandBase.java:808) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:900) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1236) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.Execute(CommandBase.java:915) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:271) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:335) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:592) [engine-bll.jar:]
        at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) [:1.7.0_05-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_05-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_05-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view9.runInternalAction(Unknown Source) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VdsEventListener.masterDomainNotOperational(VdsEventListener.java:84) [engine-bll.jar:]
        at sun.reflect.GeneratedMethodAccessor546.invoke(Unknown Source) [:1.7.0_05-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_05-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_05-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.ovirt.engine.core.common.businessentities.IVdsEventListener$$$view6.masterDomainNotOperational(Unknown Source) [engine-common.jar:]
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.startReconstruct(IrsBrokerCommand.java:1669) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.ExecuteVDSCommand(IrsBrokerCommand.java:1651) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.ExecuteCommand(VDSCommandBase.java:60) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.Execute(VdcCommandBase.java:41) [engine-dal.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:390) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData.ProceedStoragePoolStats(IrsBrokerCommand.java:308) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData.access$200(IrsBrokerCommand.java:132) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData$1.runInTransaction(IrsBrokerCommand.java:213) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:]
        at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand$IrsProxyData._updatingTimer_Elapsed(IrsBrokerCommand.java:203) [engine-vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) [:1.7.0_05-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_05-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_05-icedtea]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:]

2012-07-25 21:01:33,297 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [cfa2e4c] Weird return value: Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         661
mMessage                      Cannot acquire host id: ('23fc84ef-2a31-4a20-a8c2-5802fea9e047', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))
Comment 1 Ayal Baron 2012-07-29 05:40:27 UTC 
Haim, where are the logs?
IS there also an issue with vdsm here? (i.e. why is there an invalid argument to sanlock)?
Comment 2 Haim 2012-07-29 10:30:26 UTC 
Created attachment 601008 [details]
engine.log

attachment probably failed, there is a new reproduction on SI12.
Comment 3 Ayal Baron 2012-07-29 12:26:39 UTC 
what about vdsm.log?
sanlock log?

You did not answer my question about vdsm.
Comment 4 Itamar Heim 2012-07-30 10:12:16 UTC 
*** Bug 844315 has been marked as a duplicate of this bug. ***
Comment 7 Maor 2012-08-07 15:14:53 UTC 
Suggested solution posted at http://gerrit.ovirt.org/#/c/6986/
Comment 8 Maor 2012-08-08 07:45:14 UTC 
Merged upstream at commit #bcb08c73aca235dcd9d2fb83c06d18598ae2f6d9
Comment 11 Dafna Ron 2012-08-23 13:46:47 UTC 
verified on si15
Note You need to log in before you can comment on or make changes to this bug.