Bug 828905

Summary: Problem when an old bundle version is deleted
Product: [Other] RHQ Project Reporter: John Mazzitelli <mazz>
Component: ProvisioningAssignee: John Mazzitelli <mazz>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: bkramer, hrupp, loleary, mazz
Target Milestone: ---   
Target Release: RHQ 4.5.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 826995 Environment:
Last Closed: 2013-09-01 10:17:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 826995, 826998, 916380    
Bug Blocks:    

Description John Mazzitelli 2012-06-05 14:56:44 UTC 
+++ This bug was initially created as a clone of Bug #826995 +++

Description of problem:
The "version_order" data in the "rhq_bundle_version" table is not updated after a version bundle is deleted. This will cause an exception to be thrown when new bundle version is deployed.

The exception from rhq-server-log4j.log file is:

2012-05-31 11:20:28,272 DEBUG [org.rhq.enterprise.server.util.CriteriaQueryGenerator] SELECT  new org.rhq.core.domain.bundle.composite.BundleWithLatestVersionComposite(    bundle.id,   bundle.name,   bundle.description,   ( SELECT bv1.version FROM bundle.bundleVersions bv1 WHERE bv1.versionOrder = (SELECT MAX(bv2.versionOrder) FROM BundleVersion bv2 WHERE bv2.bundle.id = bundle.id) ) AS latestVersion,   ( SELECT COUNT(bv3) FROM bundle.bundleVersions bv3 WHERE bv3.bundle.id = bundle.id) AS deploymentCount ) 
FROM Bundle bundle


2012-05-31 11:20:28,275 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 0, SQLState: 21000
2012-05-31 11:20:28,275 ERROR [org.hibernate.util.JDBCExceptionReporter] ERROR: more than one row returned by a subquery used as an expression
2012-05-31 11:20:28,275 DEBUG [org.rhq.enterprise.server.util.HibernatePerformanceMonitor] HibernateStats[ queries=0, xactions=0, loads=0, connects=1, time=3 ] for SLSB:BundleManagerBean.findBundlesWithLatestVersionCompositesByCriteria
2012-05-31 11:20:28,278 WARN  [gwt-log] Sending exception to client: [1338459628276] 
javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.DataException: could not execute query
	at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
	at $Proxy467.findBundlesWithLatestVersionCompositesByCriteria(Unknown Source)
	at org.rhq.enterprise.gui.coregui.server.gwt.BundleGWTServiceImpl.findBundlesWithLatestVersionCompositesByCriteria(BundleGWTServiceImpl.java:329)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at org.rhq.enterprise.gui.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:82)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:283)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
	at org.rhq.enterprise.gui.coregui.server.filter.CacheControlFilter.doFilter(CacheControlFilter.java:70)
	at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:218)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
	at sun.reflect.GeneratedMethodAccessor233.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:218)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:218)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
	at java.lang.Thread.run(Thread.java:619)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.DataException: could not execute query
	at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
	at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:73)
	at org.rhq.enterprise.server.util.CriteriaQueryRunner.getCollection(CriteriaQueryRunner.java:99)
	at org.rhq.enterprise.server.util.CriteriaQueryRunner.execute(CriteriaQueryRunner.java:69)
	at org.rhq.enterprise.server.bundle.BundleManagerBean.findBundlesWithLatestVersionCompositesByCriteria(BundleManagerBean.java:1545)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.rhq.enterprise.server.common.PerformanceMonitorInterceptor.monitorHibernatePerformance(PerformanceMonitorInterceptor.java:26)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
	at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156)
	at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
	... 97 more
Caused by: org.hibernate.exception.DataException: could not execute query
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:77)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.loader.Loader.doList(Loader.java:2216)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
	at org.hibernate.loader.Loader.list(Loader.java:2099)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
	at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
	at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
	at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
	at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
	at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:64)
	... 128 more
Caused by: org.postgresql.util.PSQLException: ERROR: more than one row returned by a subquery used as an expression
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
	at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.postgresql.ds.jdbc23.AbstractJdbc23PooledConnection$StatementHandler.invoke(AbstractJdbc23PooledConnection.java:455)
	at $Proxy50.executeQuery(Unknown Source)
	at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:90)
	at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)
	at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:187)
	at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
	at org.hibernate.loader.Loader.doQuery(Loader.java:674)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
	at org.hibernate.loader.Loader.doList(Loader.java:2213)
	... 136 more


Version-Release number of selected component (if applicable):
JON 3.0.1

How reproducible:
Always

Steps to Reproduce:
1. Deploy bundle version 1;
2. Deploy bundle version 2;
3. Delete bundle version 1;
4. Try to deploy bundle version 3;
  
Actual results:
Bundle version 3 is not deployed and exception (as above) is logged in the rhq-server-log4j.log file. In JON UI, the following error message is shown: "Failed to load bundle with the latest version data"

Expected results:
Bundle version 3 is properly deployed.

Additional info:

Here is the content of the rhq_bundle_version table after steps to reproduce:

1. Deployed bundle version 1:

select * from rhq_bundle_version
id	name	                description	version	version_order	action	config_def_id	bundle_id
10051	deployServerPassword	first deploy	1	0		10811	10041
id	name	                description	version	version_order	action	config_def_id	bundle_id
Query Time=1

2.  Deployed bundle version 2:

select * from rhq_bundle_version
id	name	                description	version	version_order	action	config_def_id	bundle_id
10051	deployServerPassword	first deploy	1	0		10811	10041
10042	deployServerPassword	first deploy	2	1		10802	10041
id	name	                description	version	version_order	action	config_def_id	bundle_id
Query Time=0

3. Deleted bundle version 1:

select * from rhq_bundle_version
id	name	                description	version	version_order	action	config_def_id	bundle_id
10042	deployServerPassword	first deploy	2	1		10802	10041
id	name	                description	version	version_order	action	config_def_id	bundle_id
Query Time=0

4. Bundle version 3 - deployed:

select * from rhq_bundle_version
id	name	                description	version	version_order	action	config_def_id	bundle_id
10042	deployServerPassword	first deploy	2	1		10802	10041
10061	deployServerPassword	first deploy	3	1		10821	10041
id	name	                description	version	version_order	action	config_def_id	bundle_id
Query Time=0

--- Additional comment from loleary on 2012-06-04 16:30:34 EDT ---

No help make this issue clear and better define, 'deploy' and 'deployed' as mentioned in comment #0 refers to the actual creation of the bundle and its versions. i.e. uploading a bundle ZIP containing version 1, then uploading a bundle ZIP that contains version 2, deleting bundle version 1, then uploading a bundle ZIP that contains version 3.

Additionally, it appears that the upload and creation of bundle version 3 is successful but nothing is displayed in the UI due to the duplicate rows as described in comment #0. This means version 3 can not be deployed to a destination because the bundle is no longer available from the UI.

However, if you create bundle version 4, the versions become visible again in the UI and the UI error goes away. Once version 4 has been created the rhq_bundle_version table looks like:

    Results for statement 1 of 1:
    SELECT name, version, version_order FROM rhq_bundle_version WHERE name = 'base_jee_test'
    name	version	version_order
    base_jee_test	1.0.1-SNAPSHOT	1
    base_jee_test	1.0.2-SNAPSHOT	1
    base_jee_test	1.0.3-SNAPSHOT	2
    name	version	version_order
    Query Time=1

--- Additional comment from loleary on 2012-06-04 16:44:07 EDT ---

To provide even further details and correct a statement from comment #1:


> ...This means version 3 can not be deployed to a destination because the bundle is no longer available from the UI....

You actually can still deploy the version to a destination. You just don't see the bundle listed on the Bundles (http://localhost:7080/coregui/#Bundles) page and you see mentioned error message in the UI. However, the navigation tree does not seem to be affected. You can still expand and select Destinations from the navigation tree and deploy the bundle via the destination page.

Not sure what other issues these duplicate version_order rows will cause. Perhaps reverting to a previous version will be broken?

--- Additional comment from mazz on 2012-06-04 17:00:59 EDT ---

for the record, there is this unit test:

org.rhq.enterprise.server.bundle.BundleManagerBeanTest.testCreateBundleVersionOrdering()

that we can add to in order to check for this condition (unfortunately, it currently does not test for the case when the bundle version is deleted, which is why we didn't catch this bug).
Comment 1 John Mazzitelli 2012-06-05 18:49:58 UTC 
git commit to master: f364f5f

this also updates the unit tests in BundleManagerBeanTest to confirm the fix.
Comment 2 Larry O'Leary 2012-06-05 19:00:38 UTC 
*** Bug 826998 has been marked as a duplicate of this bug. ***
Comment 3 John Mazzitelli 2013-02-26 18:23:54 UTC 
if you think you have this problem, run this manual SQL query to find out which bundle versions are duplicated:

select name, version_order, count(version_order)
  from rhq_bundle_version t1
  group by name, version_order
 having count(version_order) > 1;

this will tell you which bundle name has more than one bundle version with the same version_order. These are the rows you need to manually update - either delete the from the UI or do a manual SQL UPDATE to fix the rows so you have monotonically increasing version_order values starting at 0.
Comment 4 Heiko W. Rupp 2013-09-01 10:17:10 UTC 
Bulk closing of items that are on_qa and in old RHQ releases, which are out for a long time and where the issue has not been re-opened since.