Bug 1290040 - [Upgrade] Engine failed to load after upgrading 3.5->3.6
[Upgrade] Engine failed to load after upgrading 3.5->3.6
Status: CLOSED WORKSFORME
Product: ovirt-engine
Classification: oVirt
Component: Setup.Engine (Show other bugs)
3.6.1
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-3.6.2
: ---
Assigned To: Shirly Radco
Pavel Stehlik
reports
:
: 1322397 (view as bug list)
Depends On:
Blocks: RHEV3.6Upgrade
  Show dependency treegraph
 
Reported: 2015-12-09 09:41 EST by Gil Klein
Modified: 2016-04-04 05:17 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-20 15:48:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Reports
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
oourfali: ovirt‑3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
sosreport (10.26 MB, application/x-xz)
2015-12-09 09:52 EST, Gil Klein
no flags Details

  None (edit)
Description Gil Klein 2015-12-09 09:41:52 EST
Description of problem:
Engine failed to load after upgrading 3.5->3.6


Version-Release number of selected component (if applicable):
From: rhevm-reports-3.5.5-2.el6ev.noarch
To: rhevm-reports-setup-3.6.1-1.el6ev.noarch


How reproducible:
On one environment only

Steps to Reproduce:
1. Install RHEV 3.5
2. Run engine-config and setup the system
3. Add RHEV 3.6 repos and yum update
4. Run engine-config and trigger the upgrade

Actual results:
Web-admin is inaccessible after the upgrade, server.logs show errors


Expected results:
Web-admin should be accessible after the upgrade


Additional info:

2015-12-09 16:19:24,752 INFO  [org.jboss.as.controller] (DeploymentScanner-threads - 2) JBAS014774: Service status report
JBAS014775:    New missing/unsatisfied dependencies:
      service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START (missing) dependents: [service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformationStart, service jboss.deployment.subunit."engine.ear"."bll.jar".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."bll.jar".component.EventQueue.START (missing) dependents: [service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformationStart, service jboss.deployment.subunit."engine.ear"."bll.jar".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START (missing) dependents: [service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformationStart, service jboss.deployment.subunit."engine.ear"."bll.jar".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."bll.jar".component.LockManager.START (missing) dependents: [service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformationStart, service jboss.deployment.subunit."engine.ear"."bll.jar".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."bll.jar".component.VdsEventListener.START (missing) dependents: [service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformationStart, service jboss.deployment.subunit."engine.ear"."bll.jar".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."bll.jar".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformation (missing) dependents: [service jboss.deployment.subunit."engine.ear"."bll.jar".moduleDeploymentRuntimeInformationStart] 
      service jboss.deployment.subunit."engine.ear"."docs.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."root.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."scheduler.jar".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."services.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."userportal.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."webadmin.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.deployment.subunit."engine.ear"."welcome.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."engine.ear".deploymentCompleteService] 
      service jboss.module.spec.service."deployment.engine.ear".main (missing) dependents: [service jboss.module.service."deployment.restapi.war".main, service jboss.module.resolve.phase."deployment.restapi.war".main.1, service jboss.module.resolve.phase."deployment.legacy_restapi.war".main.1, service jboss.module.service."deployment.legacy_restapi.war".main] 
      service module.resolved.service."deployment.legacy_restapi.war".main (missing) dependents: [service jboss.module.service."deployment.legacy_restapi.war".main] 
      service module.resolved.service."deployment.restapi.war".main (missing) dependents: [service jboss.module.service."deployment.restapi.war".main] 
JBAS014777:   Services which failed to start:      service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START

2015-12-09 16:19:29,297 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found engine.ear in deployment directory. To trigger deployment create a file called engine.ear.dodeploy
Comment 1 Gil Klein 2015-12-09 09:52 EST
Created attachment 1103948 [details]
sosreport
Comment 2 Oved Ourfali 2015-12-10 02:11:18 EST
Martin - please take a look at this one.
Gil - is putting an empty "engine.ear.dodeploy" file in the deployments directory fixes the issue?
Comment 3 Martin Perina 2015-12-10 03:01:57 EST
I found two issues:

1. engine-setup issue
   According to /var/log/ovirt-engine/setup/ovirt-engine-setup-20151208175005-p5xtuv.log
   there was an issue while upgrading reports:

      ERROR: relation "jidashboardmodel" does not exist
      Position: 4347
      com.jaspersoft.jasperserver.api.JSExceptionWrapper: could not execute query; SQL

  So engine-setup finished unsuccessfully.

2. Incorrect vsdm-jsonrpc-java version
   According to /var/log/ovirt-engine/server.log we cannot find class
   org/ovirt/vdsm/jsonrpc/client/utils/retry/RetryPolicy
Comment 4 Oved Ourfali 2015-12-10 03:16:04 EST
(In reply to Martin Perina from comment #3)
> I found two issues:
> 
> 1. engine-setup issue
>    According to
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20151208175005-p5xtuv.log
>    there was an issue while upgrading reports:
> 
>       ERROR: relation "jidashboardmodel" does not exist
>       Position: 4347
>       com.jaspersoft.jasperserver.api.JSExceptionWrapper: could not execute
> query; SQL
> 
>   So engine-setup finished unsuccessfully.
> 

Shirly, can you take a look?

> 2. Incorrect vsdm-jsonrpc-java version
>    According to /var/log/ovirt-engine/server.log we cannot find class
>    org/ovirt/vdsm/jsonrpc/client/utils/retry/RetryPolicy

Piotr, can you take a look?
Comment 5 Gil Klein 2015-12-10 03:37:12 EST
Oved, Adding an empty "engine.ear.dodeploy" file, did not help resolving this.

# mv /var/lib/ovirt-engine/jboss_runtime/deployments/engine.ear.failed /var/lib/ovirt-engine/jboss_runtime/deployments/engine.ear.failed.bak
# touch /var/lib/ovirt-engine/jboss_runtime/deployments/engine.ear.dodeploy
# /etc/init.d/ovirt-engine stop
Stopping oVirt Engine:                                     [  OK  ]
# /etc/init.d/ovirt-engine start
Starting oVirt Engine:                                     [  OK  ]
# tail -20 server.log 
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb.jar:7.5.12.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_91]

2015-12-10 10:30:03,603 ERROR [org.jboss.as.txn] (ajp-/127.0.0.1:8702-1) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException
	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
	at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)
	at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)
	at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web.jar:7.5.5.Final-redhat-3]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:189) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb.jar:7.5.12.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_91]

2015-12-10 10:30:07,525 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015003: Found engine.ear in deployment directory. To trigger deployment create a file called engine.ear.dodeploy
Comment 6 Oved Ourfali 2015-12-10 03:47:59 EST
If engine setup finished unsuccesfully, then I guess the assumption that the engine should start is wrong.
Gil?
Comment 7 Piotr Kliczewski 2015-12-10 03:49:27 EST
(In reply to Oved Ourfali from comment #4)
> (In reply to Martin Perina from comment #3)
> 
> > 2. Incorrect vsdm-jsonrpc-java version
> >    According to /var/log/ovirt-engine/server.log we cannot find class
> >    org/ovirt/vdsm/jsonrpc/client/utils/retry/RetryPolicy
> 
> Piotr, can you take a look?

This is interesting because missing class is available in 1.0.15 but when we upgrade the engine there is 1.1.5 which do not contain it. I wonder why 3.6.1 expects older class. Were all the artifacts upgraded correctly?
Comment 8 Gil Klein 2015-12-10 04:00:26 EST
(In reply to Oved Ourfali from comment #6)
> If engine setup finished unsuccesfully, then I guess the assumption that the
> engine should start is wrong.
> Gil?
I agree, I can not assume anything in that case.

Is there anyway to confirm, if any engine specific upgrade tasks should run after the upgrade of reports?
Comment 9 Oved Ourfali 2015-12-10 08:03:00 EST
Moving the bug to reports, to examine what happened there.
Gil - if it doesn't happen on succesful installations, then I see no infra issue here. Either integration or reports, depends on the issue.
If you experience issues after succesful installation, please move it back to me.
Comment 10 Sandro Bonazzola 2015-12-14 04:44:29 EST
Didi can you work with Shirly on this?
Comment 11 Yedidyah Bar David 2015-12-15 04:15:25 EST
The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is anything else here, or can be closed as duplicate of it.
Comment 12 Gil Klein 2015-12-15 04:36:09 EST
(In reply to Yedidyah Bar David from comment #11)
> The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is
> anything else here, or can be closed as duplicate of it.
Can we just confirm comment #8, before closing as duplicate, just to be sure nothing else hides here?
Comment 13 Yedidyah Bar David 2015-12-15 04:55:21 EST
(In reply to Gil Klein from comment #12)
> (In reply to Yedidyah Bar David from comment #11)
> > The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is
> > anything else here, or can be closed as duplicate of it.
> Can we just confirm comment #8, before closing as duplicate, just to be sure
> nothing else hides here?

Not sure what you mean:


(In reply to Gil Klein from comment #8)
> (In reply to Oved Ourfali from comment #6)
> > If engine setup finished unsuccesfully, then I guess the assumption that the
> > engine should start is wrong.
> > Gil?
> I agree, I can not assume anything in that case.
> 
> Is there anyway to confirm, if any engine specific upgrade tasks should run
> after the upgrade of reports?

'engine-setup' should be enough.

In general we want to make engine-setup cleanly rollback. We try quite hard, although we definitely do not provide 100% guarantee.

In a failed upgrade of 3.5 to 3.6, the 3.5 version should be startable, but will not automatically be started by engine-setup after the rollback. If you have a specific flow that does not cleanly rollback, and you think it should be fixed, please change summary accordingly and provide flow. Thanks :-) Otherwise I think we can close.
Comment 14 Gil Klein 2015-12-20 15:48:03 EST
(In reply to Yedidyah Bar David from comment #13)
> (In reply to Gil Klein from comment #12)
> > (In reply to Yedidyah Bar David from comment #11)
> > > The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is
> > > anything else here, or can be closed as duplicate of it.
> > Can we just confirm comment #8, before closing as duplicate, just to be sure
> > nothing else hides here?
> 
> Not sure what you mean:
> 
> 
> (In reply to Gil Klein from comment #8)
> > (In reply to Oved Ourfali from comment #6)
> > > If engine setup finished unsuccesfully, then I guess the assumption that the
> > > engine should start is wrong.
> > > Gil?
> > I agree, I can not assume anything in that case.
> > 
> > Is there anyway to confirm, if any engine specific upgrade tasks should run
> > after the upgrade of reports?
> 
> 'engine-setup' should be enough.
> 
> In general we want to make engine-setup cleanly rollback. We try quite hard,
> although we definitely do not provide 100% guarantee.
> 
> In a failed upgrade of 3.5 to 3.6, the 3.5 version should be startable, but
> will not automatically be started by engine-setup after the rollback. If you
> have a specific flow that does not cleanly rollback, and you think it should
> be fixed, please change summary accordingly and provide flow. Thanks :-)
> Otherwise I think we can close.
I've tried several times, but I was not able to reproduce it again. Will reopen in case I'll see it again.
Comment 15 Yedidyah Bar David 2016-04-04 05:17:35 EDT
*** Bug 1322397 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.