Created attachment 1103948 [details]
sosreport

Martin - please take a look at this one.
Gil - is putting an empty "engine.ear.dodeploy" file in the deployments directory fixes the issue?

I found two issues:

1. engine-setup issue
   According to /var/log/ovirt-engine/setup/ovirt-engine-setup-20151208175005-p5xtuv.log
   there was an issue while upgrading reports:

      ERROR: relation "jidashboardmodel" does not exist
      Position: 4347
      com.jaspersoft.jasperserver.api.JSExceptionWrapper: could not execute query; SQL

  So engine-setup finished unsuccessfully.

2. Incorrect vsdm-jsonrpc-java version
   According to /var/log/ovirt-engine/server.log we cannot find class
   org/ovirt/vdsm/jsonrpc/client/utils/retry/RetryPolicy

(In reply to Martin Perina from comment #3)
> I found two issues:
> 
> 1. engine-setup issue
>    According to
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20151208175005-p5xtuv.log
>    there was an issue while upgrading reports:
> 
>       ERROR: relation "jidashboardmodel" does not exist
>       Position: 4347
>       com.jaspersoft.jasperserver.api.JSExceptionWrapper: could not execute
> query; SQL
> 
>   So engine-setup finished unsuccessfully.
> 

Shirly, can you take a look?

> 2. Incorrect vsdm-jsonrpc-java version
>    According to /var/log/ovirt-engine/server.log we cannot find class
>    org/ovirt/vdsm/jsonrpc/client/utils/retry/RetryPolicy

Piotr, can you take a look?

Oved, Adding an empty "engine.ear.dodeploy" file, did not help resolving this.

# mv /var/lib/ovirt-engine/jboss_runtime/deployments/engine.ear.failed /var/lib/ovirt-engine/jboss_runtime/deployments/engine.ear.failed.bak
# touch /var/lib/ovirt-engine/jboss_runtime/deployments/engine.ear.dodeploy
# /etc/init.d/ovirt-engine stop
Stopping oVirt Engine:                                     [  OK  ]
# /etc/init.d/ovirt-engine start
Starting oVirt Engine:                                     [  OK  ]
# tail -20 server.log 
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb.jar:7.5.12.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_91]

2015-12-10 10:30:03,603 ERROR [org.jboss.as.txn] (ajp-/127.0.0.1:8702-1) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException
	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
	at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)
	at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)
	at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web.jar:7.5.5.Final-redhat-3]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:189) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb.jar:7.5.12.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb.jar:7.5.12.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_91]

2015-12-10 10:30:07,525 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015003: Found engine.ear in deployment directory. To trigger deployment create a file called engine.ear.dodeploy

If engine setup finished unsuccesfully, then I guess the assumption that the engine should start is wrong.
Gil?

(In reply to Oved Ourfali from comment #4)
> (In reply to Martin Perina from comment #3)
> 
> > 2. Incorrect vsdm-jsonrpc-java version
> >    According to /var/log/ovirt-engine/server.log we cannot find class
> >    org/ovirt/vdsm/jsonrpc/client/utils/retry/RetryPolicy
> 
> Piotr, can you take a look?

This is interesting because missing class is available in 1.0.15 but when we upgrade the engine there is 1.1.5 which do not contain it. I wonder why 3.6.1 expects older class. Were all the artifacts upgraded correctly?

(In reply to Oved Ourfali from comment #6)
> If engine setup finished unsuccesfully, then I guess the assumption that the
> engine should start is wrong.
> Gil?
I agree, I can not assume anything in that case.

Is there anyway to confirm, if any engine specific upgrade tasks should run after the upgrade of reports?

Moving the bug to reports, to examine what happened there.
Gil - if it doesn't happen on succesful installations, then I see no infra issue here. Either integration or reports, depends on the issue.
If you experience issues after succesful installation, please move it back to me.

Didi can you work with Shirly on this?

The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is anything else here, or can be closed as duplicate of it.

(In reply to Yedidyah Bar David from comment #11)
> The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is
> anything else here, or can be closed as duplicate of it.
Can we just confirm comment #8, before closing as duplicate, just to be sure nothing else hides here?

(In reply to Gil Klein from comment #12)
> (In reply to Yedidyah Bar David from comment #11)
> > The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is
> > anything else here, or can be closed as duplicate of it.
> Can we just confirm comment #8, before closing as duplicate, just to be sure
> nothing else hides here?

Not sure what you mean:


(In reply to Gil Klein from comment #8)
> (In reply to Oved Ourfali from comment #6)
> > If engine setup finished unsuccesfully, then I guess the assumption that the
> > engine should start is wrong.
> > Gil?
> I agree, I can not assume anything in that case.
> 
> Is there anyway to confirm, if any engine specific upgrade tasks should run
> after the upgrade of reports?

'engine-setup' should be enough.

In general we want to make engine-setup cleanly rollback. We try quite hard, although we definitely do not provide 100% guarantee.

In a failed upgrade of 3.5 to 3.6, the 3.5 version should be startable, but will not automatically be started by engine-setup after the rollback. If you have a specific flow that does not cleanly rollback, and you think it should be fixed, please change summary accordingly and provide flow. Thanks :-) Otherwise I think we can close.

(In reply to Yedidyah Bar David from comment #13)
> (In reply to Gil Klein from comment #12)
> > (In reply to Yedidyah Bar David from comment #11)
> > > The jidashboardmodel issue is tracked in bug 1289660. Not sure if there is
> > > anything else here, or can be closed as duplicate of it.
> > Can we just confirm comment #8, before closing as duplicate, just to be sure
> > nothing else hides here?
> 
> Not sure what you mean:
> 
> 
> (In reply to Gil Klein from comment #8)
> > (In reply to Oved Ourfali from comment #6)
> > > If engine setup finished unsuccesfully, then I guess the assumption that the
> > > engine should start is wrong.
> > > Gil?
> > I agree, I can not assume anything in that case.
> > 
> > Is there anyway to confirm, if any engine specific upgrade tasks should run
> > after the upgrade of reports?
> 
> 'engine-setup' should be enough.
> 
> In general we want to make engine-setup cleanly rollback. We try quite hard,
> although we definitely do not provide 100% guarantee.
> 
> In a failed upgrade of 3.5 to 3.6, the 3.5 version should be startable, but
> will not automatically be started by engine-setup after the rollback. If you
> have a specific flow that does not cleanly rollback, and you think it should
> be fixed, please change summary accordingly and provide flow. Thanks :-)
> Otherwise I think we can close.
I've tried several times, but I was not able to reproduce it again. Will reopen in case I'll see it again.

*** Bug 1322397 has been marked as a duplicate of this bug. ***