Bug 1244718

Summary:	Engine does not start if a non-responsive host with running VM exists in the DB
Product:	[oVirt] ovirt-engine	Reporter:	Yevgeny Zaspitsky <yzaspits>
Component:	General	Assignee:	Yevgeny Zaspitsky <yzaspits>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Karolína Hajná <khajna>
Severity:	high	Docs Contact:
Priority:	high
Version:	---	CC:	bazulay, bugs, dfediuck, ecohen, gklein, lsurette, masayag, pstehlik, rbalakri, rgolan, yeylon, yzaspits
Target Milestone:	ovirt-3.6.0-rc	Flags:	rule-engine: ovirt-3.6.0+ ylavi: planning_ack+ rule-engine: devel_ack+ pnovotny: testing_ack+
Target Release:	3.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	infra
Fixed In Version:	3.6.0-4_alpha3	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-04 11:48:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yevgeny Zaspitsky 2015-07-20 11:06:05 UTC

Description of problem:
Engine does not start if a non-responsive host with running VM exists in the DB

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create new DC, cluster and add a new host
2. Run a VM in the new created cluster that will run on the new added host.
3. After the VM runs, make the host non-responsive (e.g shut vdsm service down on the host)
4. Restart engine

Actual results:
The engine does not start.

java.lang.IllegalStateException: JBAS014232: @PostConstruct method of EJB singleton Backend of type org.ovirt.engine.core.bll.Backend has been recursively invoked

Expected results:
Engine starts with no exceptions

Additional info:

2015-07-19 18:26:57,590 ERROR [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 46) [] Error during initialization: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke private void org.ovirt.engine.core.vdsbroker.ResourceManager.init() on org.ovirt.engine.core.vdsbroker.ResourceManager@62453f10
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.producer.BeanInjectionTarget.postConstruct(BeanInjectionTarget.java:70) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:153) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:840) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:370) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:381) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:121) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:150) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:762) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:168) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:88) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.ovirt.engine.core.bll.Backend.loadService(Backend.java:283) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.initialize(Backend.java:271) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.create(Backend.java:145) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:407)
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:117) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:407)
	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:55) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ee.component.ComponentInstantiatorInterceptor.processInvocation(ComponentInstantiatorInterceptor.java:75)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:94) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:260) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:399) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43) [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:160)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:133)
	at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:89)
	at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:137) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	... 82 more
Caused by: javax.ejb.EJBException: java.lang.IllegalStateException: JBAS014232: @PostConstruct method of EJB singleton Backend of type org.ovirt.engine.core.bll.Backend has been recursively invoked
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:262) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:399) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:95) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:448)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view5.getResourceManager(Unknown Source) [common.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:414) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:65) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
	at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.getResourceManager(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.VdsEventListener.removeAsyncRunningCommand(VdsEventListener.java:490) [bll.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager.RemoveAsyncRunningVm(ResourceManager.java:202) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager.SetVmUnknown(ResourceManager.java:319) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager.init(ResourceManager.java:156) [vdsbroker.jar:]
	... 87 more
Caused by: java.lang.IllegalStateException: JBAS014232: @PostConstruct method of EJB singleton Backend of type org.ovirt.engine.core.bll.Backend has been recursively invoked
	at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:119) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:47) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:260) [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
	... 132 more

Comment 1 Yevgeny Zaspitsky 2015-07-20 15:29:29 UTC

Correction for reproduction method:
3. When the VM starts up, make the host non-responsive

Comment 2 Karolína Hajná 2015-10-22 09:02:27 UTC

Verified on 3.6.0.1-1

Comment 3 Sandro Bonazzola 2015-11-04 11:48:17 UTC

oVirt 3.6.0 has been released on November 4th, 2015 and should fix this issue.
If problems still persist, please open a new BZ and reference this one.