Bug 949694 - rhevm-upgrade is failing between si26.4 to si27.4 (3.1.3) in async task cleanup
Summary: rhevm-upgrade is failing between si26.4 to si27.4 (3.1.3) in async task cleanup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup
Version: 3.1.3
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ---
: 3.1.4
Assignee: Alex Lourie
QA Contact: Ilanit Stein
URL:
Whiteboard: infra
Depends On: 921201 921202
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-04-08 19:04 UTC by Idith Tal-Kohen
Modified: 2018-12-01 15:47 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Upgrading the Manager failed when there were leftover compensation entries during the async tasks cleanup. These entries caused the backend to abort when trying to activate the compensation procedure on system startup. The taskcleaner utility now uses the -A flag, which enables rhevm-upgrade to remove all compensation data from the business_entity_snapshot table during upgrade.
Clone Of: 921201
Environment:
Last Closed: 2013-05-01 18:35:47 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
engine log (19.35 KB, application/x-gzip)
2013-04-24 12:25 UTC, Ilanit Stein
no flags Details
vdsm log (81.27 KB, application/x-gzip)
2013-04-24 12:28 UTC, Ilanit Stein
no flags Details
Older vdsm log (that include the time in which upgrade stopped async tasks. (627.09 KB, application/x-xz)
2013-04-24 12:30 UTC, Ilanit Stein
no flags Details
si26.1->si28 engine log (15.77 KB, application/x-gzip)
2013-04-25 09:05 UTC, Ilanit Stein
no flags Details
si26.1->si28 vdsm log (304.67 KB, application/x-gzip)
2013-04-25 09:09 UTC, Ilanit Stein
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:0773 0 normal SHIPPED_LIVE rhevm 3.1.4 bug fix update 2013-05-01 22:34:02 UTC
oVirt gerrit 13567 0 None None None Never

Comment 2 Yair Zaslavsky 2013-04-18 09:41:34 UTC
https://gerrit.eng.lab.tlv.redhat.com/gitweb?p=ovirt-engine.git;a=commit;h=ae3b3c9e03b0de92454edd58d3f83e680c9292ae - fix for utility script + stored procedures.

Comment 4 Ilanit Stein 2013-04-24 12:14:19 UTC
Checked si26.1 upgrade to si27.4 (si26.4 do not exist).

Scenrio:
=======
On si26.1:
1. Created a VM, and then created a template (started aync task).
2. While template creation running, stopped ovirt-engine service
3. Run "yum update rhevm-setup" (to si27.4)
4. Run "rhevm-upgrade"  (to si27.4)

Results:
=======
The output of the rhevm-upgrade indeed different. No repeated question regarding stop async tasks, but, 
Got an exception Errors on engine.log during clear tasks by upgrade process.
On GUI, VM remained in lock state (At least for 30 min, that I've checked).
Restart vdsmd service, and then restart ovirt engine didn't change the VM lock on GUI.
  
Exception Errors on engine.log during clear tasks by upgrade process:
=====================================================================
2013-04-24 02:29:07,529 ERROR [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-27) EntityAsyncTask::EndCommandAction [within thread]: EndAction for action type AddVmTemplate threw an exception: org.springframework.dao.DataIntegrityViolationException: CallableStatementCallback; SQL [{call deletevmtemplates(?)}]; ERROR: update or delete on table "vm_static" violates foreign key constraint "vm_templates_vm_static" on table "vm_static"
  Detail: Key (vm_guid)=(00000000-0000-0000-0000-000000000000) is still referenced from table "vm_static".
  Where: SQL statement "DELETE FROM vm_static WHERE vm_guid =  $1  AND entity_type = 'TEMPLATE'"
PL/pgSQL function "deletevmtemplates" line 7 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: update or delete on table "vm_static" violates foreign key constraint "vm_templates_vm_static" on table "vm_static"
  Detail: Key (vm_guid)=(00000000-0000-0000-0000-000000000000) is still referenced from table "vm_static".
  Where: SQL statement "DELETE FROM vm_static WHERE vm_guid =  $1  AND entity_type = 'TEMPLATE'"
PL/pgSQL function "deletevmtemplates" line 7 at SQL statement
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:245) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]


Console output rhevm upgrade, no errors seen:
=============================================

[root@istein-31 yum.repos.d]# yum update rhevm-setup
Loaded plugins: product-id, rhnplugin, subscription-manager, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package rhevm-setup.noarch 0:3.1.0-43.el6ev will be updated
---> Package rhevm-setup.noarch 0:3.1.0-50.el6ev will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch          Version                  Repository      Size
================================================================================
Updating:
 rhevm-setup        noarch        3.1.0-50.el6ev           upgrade        950 k

Transaction Summary
================================================================================
Upgrade       1 Package(s)

Total download size: 950 k
Is this ok [y/N]: y
Downloading Packages:
rhevm-setup-3.1.0-50.el6ev.noarch.rpm                    | 950 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : rhevm-setup-3.1.0-50.el6ev.noarch                            1/2 
  Cleanup    : rhevm-setup-3.1.0-43.el6ev.noarch                            2/2 
  Verifying  : rhevm-setup-3.1.0-50.el6ev.noarch                            1/2 
  Verifying  : rhevm-setup-3.1.0-43.el6ev.noarch                            2/2 

Updated:
  rhevm-setup.noarch 0:3.1.0-50.el6ev                                           

Complete!
[root@istein-31 yum.repos.d]# rhevm-upgrade
Loaded plugins: product-id, rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.

Checking for updates... (This may take several minutes)
11 Updates available:
 * rhevm-3.1.0-50.el6ev.noarch
 * rhevm-backend-3.1.0-50.el6ev.noarch
 * rhevm-config-3.1.0-50.el6ev.noarch
 * rhevm-dbscripts-3.1.0-50.el6ev.noarch
 * rhevm-genericapi-3.1.0-50.el6ev.noarch
 * rhevm-notification-service-3.1.0-50.el6ev.noarch
 * rhevm-restapi-3.1.0-50.el6ev.noarch
 * rhevm-tools-common-3.1.0-50.el6ev.noarch
 * rhevm-userportal-3.1.0-50.el6ev.noarch
 * rhevm-webadmin-portal-3.1.0-50.el6ev.noarch
 * vdsm-bootstrap-4.10.2-1.8.el6ev.noarch
Stopping ovirt-engine service...                         [ DONE ]
Stopping DB related services...                          [ DONE ]
Cleaning async tasks...                                  [ DONE ]

Info: The following tasks have been found running in the system: 

System Tasks:
-----
Task ID: 201
Task name: AddVmTemplateCommand
Task description: Adding a template




[ Apr 24 02:27:19 ] Would you like to proceed and try to stop tasks automatically?
(Answering 'no' will stop the upgrade)? (yes|no): yes

[ Apr 24 02:27:59 ] System will try to clear tasks during the next 3 minutes.



Pre-upgrade validations...                               [ DONE ]
Backing Up Database...                                   [ DONE ]
Rename Database...                                       [ DONE ]
Updating rpms (This may take a while
depending on your connection speed to the repository.
To track the installation process run tail -f /var/log/yum.log in another session).
Note that the transition from package removal to package installation is expected
to take a significant amount of time. No output will appear in the log file during
this transition.... [ DONE ]
Updating Database...                                     [ DONE ]
Restore Database name...                                 [ DONE ]
Running post install configuration...                    [ DONE ]
Starting ovirt-engine...                                 [ DONE ]

RHEV Manager upgrade completed successfully!

* Upgrade log available at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_04_24_02_26_34.log
* DB Backup available at  /var/lib/ovirt-engine/backups/ovirt-engine_db_backup_2013_04_24_02_26_35.sql
[root@istein-31 yum.repos.d]# 
[root@istein-31 yum.repos.d]# 
[root@istein-31 yum.repos.d]# rpm -q rhevm
rhevm-3.1.0-50.el6ev.noarch
[root@istein-31 yum.repos.d]#

Comment 5 Ilanit Stein 2013-04-24 12:25:00 UTC
Created attachment 739442 [details]
engine log

Comment 6 Ilanit Stein 2013-04-24 12:28:22 UTC
Created attachment 739444 [details]
vdsm log

Comment 7 Ilanit Stein 2013-04-24 12:30:25 UTC
Created attachment 739446 [details]
Older vdsm log (that include the time in which upgrade stopped async tasks.

Comment 9 Ilanit Stein 2013-04-25 08:58:41 UTC
Retested for si26.1 (3.1.1) -> si28 (3.14) upgrade:
Result same as comment 4. new engine/vdsm logs attached.

Cosole log:
==========
[root@istein-31 yum.repos.d]# yum update rhevm-setup
Loaded plugins: product-id, rhnplugin, subscription-manager, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
JBOSS                                                                                                                                                                         | 1.9 kB     00:00     
qa-latest                                                                                                                                                                     | 1.3 kB     00:00     
rhel                                                                                                                                                                          | 3.9 kB     00:00     
rhel-optional                                                                                                                                                                 | 3.7 kB     00:00     
rhevm_deps_acceptance_repo                                                                                                                                                    | 1.3 kB     00:00     
upgrade                                                                                                                                                                       | 1.3 kB     00:00     
upgrade/primary                                                                                                                                                               |  14 kB     00:00     
upgrade                                                                                                                                                                                        65/65
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package rhevm-setup.noarch 0:3.1.0-43.el6ev will be updated
---> Package rhevm-setup.noarch 0:3.1.0-52.el6ev will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================================================
 Package                                          Arch                                        Version                                             Repository                                    Size
=====================================================================================================================================================================================================
Updating:
 rhevm-setup                                      noarch                                      3.1.0-52.el6ev                                      upgrade                                      952 k

Transaction Summary
=====================================================================================================================================================================================================
Upgrade       1 Package(s)

Total download size: 952 k
Is this ok [y/N]: y
Downloading Packages:
rhevm-setup-3.1.0-52.el6ev.noarch.rpm                                                                                                                                         | 952 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : rhevm-setup-3.1.0-52.el6ev.noarch                                                                                                                                                 1/2 
  Cleanup    : rhevm-setup-3.1.0-43.el6ev.noarch                                                                                                                                                 2/2 
  Verifying  : rhevm-setup-3.1.0-52.el6ev.noarch                                                                                                                                                 1/2 
  Verifying  : rhevm-setup-3.1.0-43.el6ev.noarch                                                                                                                                                 2/2 

Updated:
  rhevm-setup.noarch 0:3.1.0-52.el6ev                                                                                                                                                
[root@istein-31 yum.repos.d]# rhevm-upgrade
Loaded plugins: product-id, rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.

Checking for updates... (This may take several minutes)
11 Updates available:
 * rhevm-3.1.0-52.el6ev.noarch
 * rhevm-backend-3.1.0-52.el6ev.noarch
 * rhevm-config-3.1.0-52.el6ev.noarch
 * rhevm-dbscripts-3.1.0-52.el6ev.noarch
 * rhevm-genericapi-3.1.0-52.el6ev.noarch
 * rhevm-notification-service-3.1.0-52.el6ev.noarch
 * rhevm-restapi-3.1.0-52.el6ev.noarch
 * rhevm-tools-common-3.1.0-52.el6ev.noarch
 * rhevm-userportal-3.1.0-52.el6ev.noarch
 * rhevm-webadmin-portal-3.1.0-52.el6ev.noarch
 * vdsm-bootstrap-4.10.2-1.10.el6ev.noarch

During the upgrade process, RHEV Manager  will not be accessible.
All existing running virtual machines will continue but you will not be able to
start or stop any new virtual machines during the process.

Would you like to proceed? (yes|no): yes
Stopping ovirt-engine service...                         [ DONE ]
Stopping DB related services...                          [ DONE ]
Cleaning async tasks...                                  [ DONE ]

Info: The following tasks have been found running in the system: 

System Tasks:
-----
Task ID: 201
Task name: AddVmTemplateCommand
Task description: Adding a template




[ Apr 24 23:41:50 ] Would you like to proceed and try to stop tasks automatically?
(Answering 'no' will stop the upgrade)? (yes|no): yes

[ Apr 24 23:41:59 ] System will try to clear tasks during the next 3 minutes.

Pre-upgrade validations...                               [ DONE ]
Backing Up Database...                                   [ DONE ]
Rename Database...                                       [ DONE ]
Updating rpms (This may take a while
depending on your connection speed to the repository.
To track the installation process run tail -f /var/log/yum.log in another session).
Note that the transition from package removal to package installation is expected
to take a significant amount of time. No output will appear in the log file during
this transition.... [ DONE ]
Updating Database...                                     [ DONE ]
Restore Database name...                                 [ DONE ]
Running post install configuration...                    [ DONE ]
Starting ovirt-engine...                                 [ DONE ]

RHEV Manager upgrade completed successfully!

* Upgrade log available at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_04_24_23_39_52.log
* DB Backup available at  /var/lib/ovirt-engine/backups/ovirt-engine_db_backup_2013_04_24_23_39_54.sql
 
[root@istein-31 yum.repos.d]# rpm -q rhevm
rhevm-3.1.0-52.el6ev.noarch

Comment 10 Ilanit Stein 2013-04-25 09:05:28 UTC
Created attachment 739744 [details]
si26.1->si28 engine log

Comment 11 Ilanit Stein 2013-04-25 09:09:04 UTC
Created attachment 739749 [details]
si26.1->si28 vdsm log

Comment 14 Barak 2013-04-25 19:31:35 UTC
I guess it failedQA because of this exception:

2013-04-24 23:43:00,025 ERROR [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-17) EntityAsyncTask::EndCommandAction [within thread]: EndAction for action type AddVmTemplate threw an exception: java.lang.NullPointerException
	at org.ovirt.engine.core.bll.VmHandler.UnLockVm(VmHandler.java:238) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.AddVmTemplateCommand.endUnlockOps(AddVmTemplateCommand.java:394) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.AddVmTemplateCommand.endDefaultOperations(AddVmTemplateCommand.java:384) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.AddVmTemplateCommand.endSuccessfully(AddVmTemplateCommand.java:370) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:512) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:459) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1374) [engine-bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:142) [engine-utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:109) [engine-utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:405) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:369) [engine-bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
	at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
	at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
	at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.endAction(Unknown Source) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.EntityAsyncTask.EndCommandAction(EntityAsyncTask.java:135) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.EntityAsyncTask.access$000(EntityAsyncTask.java:23) [engine-bll.jar:]
	at org.ovirt.engine.core.bll.EntityAsyncTask$1.run(EntityAsyncTask.java:104) [engine-bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]



Ilanit what was the upgrade/reproduce you did ?
Was it according to the steps described in comment #4 ?

Why did you restart the engine prior to the upgrade of the engine ?

Comment 15 Ilanit Stein 2013-04-28 11:25:34 UTC
Turning this bug to on QA, as it turns this bug is upgrade fail when there are  compensation entries. 

It will be checked whether there is a need to open a separate bug regaridng the exception in comment 14.

Comment 16 Ilanit Stein 2013-04-28 12:24:46 UTC
Verification flow:
=================
1. Clean install si26.1 (si26.4 do not exist)
2. Insert entry to compensation table, as it is difficult to create a "real" entry in this table.
3. Run upgrade to si28 (3.1.4)
4. Verify no upgrade successful
5. Verify the compensation table was cleared by upgrade.

Console log:
===========
[root@istein-31 yum.repos.d]# psql -U postgres engine
Password for user postgres: 
psql (8.4.13)
Type "help" for help.

engine=# select * from business_entity_snapshot ;
 id | command_id | command_type | entity_id | entity_type | entity_snapshot | snapshot_class | snapshot_type | insertion_order | started_at 
----+------------+--------------+-----------+-------------+-----------------+----------------+---------------+-----------------+------------
(0 rows)

engine=# insert into business_entity_snapshot (id,command_id,command_type) values (uuid_generate_v1(), uuid_generate_v1(), 'aaa');
INSERT 170447 1
engine=# select * from business_entity_snapshot ;
                  id                  |              command_id              | command_type | entity_id | entity_type | entity_snapshot | snapshot_class | snapshot_type | insertion_order |          started_at           
--------------------------------------+--------------------------------------+--------------+-----------+-------------+-----------------+----------------+---------------+-----------------+-------------------------------
 7efec230-af97-11e2-9cec-2f9e4a8784d5 | 7efec3e8-af97-11e2-9c32-efb26df1ff85 | aaa          |           |             |                 |                |               |                 | 2013-04-28 03:06:38.007229+03
(1 row)

engine=# \q
[root@istein-31 yum.repos.d]# vi rhevm.repo 
[root@istein-31 yum.repos.d]# yum update rhevm-setup
Loaded plugins: product-id, rhnplugin, subscription-manager, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
JBOSS                                                                                                                                                                                                                 | 1.9 kB     00:00     
qa-latest                                                                                                                                                                                                             | 1.3 kB     00:00     
rhel                                                                                                                                                                                                                  | 3.9 kB     00:00     
rhel-optional                                                                                                                                                                                                         | 3.7 kB     00:00     
rhevm_deps_acceptance_repo                                                                                                                                                                                            | 1.3 kB     00:00     
upgrade                                                                                                                                                                                                               | 1.3 kB     00:00     
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package rhevm-setup.noarch 0:3.1.0-43.el6ev will be updated
---> Package rhevm-setup.noarch 0:3.1.0-52.el6ev will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                                    Arch                                                  Version                                                       Repository                                              Size
=============================================================================================================================================================================================================================================
Updating:
 rhevm-setup                                                noarch                                                3.1.0-52.el6ev                                                upgrade                                                952 k

Transaction Summary
=============================================================================================================================================================================================================================================
Upgrade       1 Package(s)

Total download size: 952 k
Is this ok [y/N]: y
Downloading Packages:
rhevm-setup-3.1.0-52.el6ev.noarch.rpm                                                                                                                                                                                 | 952 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : rhevm-setup-3.1.0-52.el6ev.noarch                                                                                                                                                                                         1/2 
  Cleanup    : rhevm-setup-3.1.0-43.el6ev.noarch                                                                                                                                                                                         2/2 
  Verifying  : rhevm-setup-3.1.0-52.el6ev.noarch                                                                                                                                                                                         1/2 
  Verifying  : rhevm-setup-3.1.0-43.el6ev.noarch                                                                                                                                                                                         2/2 

Updated:
  rhevm-setup.noarch 0:3.1.0-52.el6ev                                                                                                                                                                                                        

Complete!
[root@istein-31 yum.repos.d]# rhevm-upgrade
Loaded plugins: product-id, rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.

Checking for updates... (This may take several minutes)
11 Updates available:
 * rhevm-3.1.0-52.el6ev.noarch
 * rhevm-backend-3.1.0-52.el6ev.noarch
 * rhevm-config-3.1.0-52.el6ev.noarch
 * rhevm-dbscripts-3.1.0-52.el6ev.noarch
 * rhevm-genericapi-3.1.0-52.el6ev.noarch
 * rhevm-notification-service-3.1.0-52.el6ev.noarch
 * rhevm-restapi-3.1.0-52.el6ev.noarch
 * rhevm-tools-common-3.1.0-52.el6ev.noarch
 * rhevm-userportal-3.1.0-52.el6ev.noarch
 * rhevm-webadmin-portal-3.1.0-52.el6ev.noarch
 * vdsm-bootstrap-4.10.2-1.10.el6ev.noarch

During the upgrade process, RHEV Manager  will not be accessible.
All existing running virtual machines will continue but you will not be able to
start or stop any new virtual machines during the process.

Would you like to proceed? (yes|no): yes
Stopping ovirt-engine service...                         [ DONE ]
Stopping DB related services...                          [ DONE ]
Cleaning async tasks...                                  [ DONE ]
Pre-upgrade validations...                               [ DONE ]
Backing Up Database...                                   [ DONE ]
Rename Database...                                       [ DONE ]
Updating rpms (This may take a while
depending on your connection speed to the repository.
To track the installation process run tail -f /var/log/yum.log in another session).
Note that the transition from package removal to package installation is expected
to take a significant amount of time. No output will appear in the log file during
this transition.... [ DONE ]
Updating Database...                                     [ DONE ]
Restore Database name...                                 [ DONE ]
Running post install configuration...                    [ DONE ]
Starting ovirt-engine...                                 [ DONE ]

RHEV Manager upgrade completed successfully!

* Upgrade log available at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_04_28_03_08_10.log
* DB Backup available at  /var/lib/ovirt-engine/backups/ovirt-engine_db_backup_2013_04_28_03_08_12.sql
[root@istein-31 yum.repos.d]# rpm -q rhevm
rhevm-3.1.0-52.el6ev.noarch
[root@istein-31 yum.repos.d]# psql -U postgres engine
Password for user postgres: 
psql (8.4.13)
Type "help" for help.

engine=# select * from business_entity_snapshot ;
 id | command_id | command_type | entity_id | entity_type | entity_snapshot | snapshot_class | snapshot_type | insertion_order | started_at 
----+------------+--------------+-----------+-------------+-----------------+----------------+---------------+-----------------+------------
(0 rows)

Comment 17 errata-xmlrpc 2013-05-01 18:35:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0773.html


Note You need to log in before you can comment on or make changes to this bug.