Bug 1508327

Summary: Engine failed to start after upgrade caused by invocation of logFreeMacs()
Product: [oVirt] ovirt-engine Reporter: Israel Pinto <ipinto>
Component: BLL.InfraAssignee: Ravi Nori <rnori>
Status: CLOSED CURRENTRELEASE QA Contact: Israel Pinto <ipinto>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.1.7.4CC: bugs, danken, lsvaty, lveyde, mavital, mperina, oourfali
Target Milestone: ovirt-4.1.7Keywords: Regression
Target Release: 4.1.7.6Flags: rule-engine: ovirt-4.1+
rule-engine: blocker+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.1.7.6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-13 12:25:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1508908    
Attachments:
Description Flags
dump and console log
none
server engine and boot logs
none
setup dir
none
debug logs none

Description Israel Pinto 2017-11-01 08:13:48 UTC
Description of problem:
After upgrading to 4.1.7.6 environment with running vms (and vm pools),
Engine failed to start. 


Version-Release number of selected component (if applicable):
4.1.7.6

How reproducible:
All the   time

Steps to Reproduce:
Upgrade engine from 4.1.7.5 to 4.1.7.6

Actual results:
Web admin not reachable 


Server log:
2017-10-31 22:19:33,028+02 INFO  [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 59) Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2017-10-31 22:19:33,028+02 INFO  [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 59) Quartz scheduler version: 2.1.2
2017-10-31 22:19:33,028+02 INFO  [org.quartz.core.QuartzScheduler] (ServerService Thread Pool -- 59) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2017-10-31 22:19:33,168+02 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 59) WFLYCLINF0002: Started timeout-base cache from ovirt-engine container
2017-10-31 22:19:39,282+02 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 59) WFLYEJB0034: EJB Invocation failed on component MacPoolPerCluster for method public void org.ovirt.engine.core.bll.network.macpool.MacPoolPerCluster.logFreeMacs(): javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) [weld-core-impl.jar:2.3.3.Final-redhat-1]
        at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
        at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
        at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.0.8.GA-redhat-1.jar:7.0.8.GA-redhat-1]
......
.....
 AND :
2017-10-31 22:19:45,831+02 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: /ovirt-engine/sso
2017-10-31 22:19:45,890+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
    Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
    Caused by: javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
    Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
    Caused by: javax.ejb.EJBException: java.lang.RuntimeException: org.ovirt.engine.core.utils.ovf.OvfReaderException: OVF error: win_12_pool-1: cannot read 'VmInit' with value: null
    Caused by: java.lang.RuntimeException: org.ovirt.engine.core.utils.ovf.OvfReaderException: OVF error: win_12_pool-1: cannot read 'VmInit' with value: null
    Caused by: org.ovirt.engine.core.utils.ovf.OvfReaderException: OVF error: win_12_pool-1: cannot read 'VmInit' with value: null"}}

Comment 1 Israel Pinto 2017-11-01 08:14:37 UTC
Created attachment 1346376 [details]
dump and console log

Comment 2 Israel Pinto 2017-11-01 08:15:24 UTC
Created attachment 1346377 [details]
server engine and boot logs

Comment 3 Israel Pinto 2017-11-01 08:16:01 UTC
Created attachment 1346378 [details]
setup dir

Comment 4 Martin Perina 2017-11-01 08:43:09 UTC
Created attachment 1346381 [details]
debug logs

Comment 5 Michal Skrivanek 2017-11-03 08:39:37 UTC
*** Bug 1508912 has been marked as a duplicate of this bug. ***

Comment 6 Israel Pinto 2017-11-05 08:16:21 UTC
Verify with:
Red Hat Virtualization Manager Version: 4.1.7.6-0.1.el7

Steps:
1. Create stateless vm/s
2. Run stateless vm
3. Restart engine

PASS - Engine is up