Bug 611456 - Delete bundle is not working
Delete bundle is not working
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Provisioning (Show other bugs)
3.0.0
All All
high Severity high (vote)
: ---
: ---
Assigned To: Jay Shaughnessy
Sunil Kondkar
:
Depends On:
Blocks: rhq_triage jon24-provisioning
  Show dependency treegraph
 
Reported: 2010-07-05 06:53 EDT by Rajan Timaniya
Modified: 2013-09-02 03:26 EDT (History)
3 users (show)

See Also:
Fixed In Version: 4.0.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-02 03:26:52 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
server log (83.05 KB, text/x-log)
2010-07-05 06:53 EDT, Rajan Timaniya
no flags Details

  None (edit)
Description Rajan Timaniya 2010-07-05 06:53:01 EDT
Created attachment 429506 [details]
server log

Description of problem:
Delete bundle is not working. It gives exceptions in server log.

2010-07-05 16:12:28,832 INFO  [STDOUT] SerialUtility.prepare [findBundleVersionsByCriteria] Detached in: 1ms, Size is: 5977
2010-07-05 16:12:48,065 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/coregui].[org.rhq.enterprise.gui.coregui.CoreGUI AlertGWTService]] Servlet.service() for servlet org.rhq.enterprise.gui.coregui.CoreGUI AlertGWTService threw exception
java.lang.RuntimeException: Failed to validate session
        at org.rhq.enterprise.gui.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:55)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.rhq.enterprise.server.auth.SessionNotFoundException
        at org.rhq.enterprise.server.auth.SessionManager.getSubject(SessionManager.java:167)
        at org.rhq.enterprise.server.auth.SubjectManagerBean.getSubjectBySessionId(SubjectManagerBean.java:568)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
        at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
        at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156)
        at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
        at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
        at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
        at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
        at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
        at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
        at $Proxy245.getSubjectBySessionId(Unknown Source)
        at org.rhq.enterprise.gui.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:53)
        ... 22 more


Version-Release number of selected component (if applicable):
JON 2.4 GA_QA (tag-jon-release build# 49)
build number: 10775:e086474

How reproducible:
Always

Steps to Reproduce:
1. Go to Administration->Content->Bundles
2. Click on button 'New'.
3. Select the radio button 'Upload'.
4. Click on the 'Browse' button.
5. Browse and select the archive file. (provide valid archive file)
6. Click on Next button and proceed for bundle creation. (complete bundle creation)
7. Select bundle and click on 'Delete' button.
  
Actual results:
Delete bundle is not working. It gives exceptions in server log.

Expected results:
Bundle should delete without exception in server log.

Additional info:
Please refer attached server log.
Comment 1 Rajan Timaniya 2010-07-06 07:13:48 EDT
Bundle deletion gives error:
	
Title :	
Failed to delete bundle [example]
Severity :	
Error
	
Time :	
Tue 06 Jul 2010 04:41:40 PM IST
Detail :
java.lang.RuntimeException: javax.ejb.EJBTransactionRolledbackException:java.lang.OutOfMemoryError: Java heap space -> java.lang.RuntimeException:java.lang.OutOfMemoryError: Java heap space -> java.lang.OutOfMemoryError:Java heap space
Comment 2 Corey Welton 2010-07-06 12:03:22 EDT
Rajan, can you provide more details as to the configuration you tested?  How large was the bundle you attempted to deploy?

Dev to open separate bug investigating possible size limitations/recommendations.
Comment 3 Rajan Timaniya 2010-07-07 05:42:42 EDT
Test environment:
Platform: RHEL 5.5
Database: Postgres 8.4
Java: Sun JDK1.6
Browser: FF 3.0.19

The bundle size was 294.0MB and it contains jboss-eap-5.0.0.GA.zip
Comment 4 Jay Shaughnessy 2010-07-08 16:57:27 EDT
The error reported here, OutOfMemory, may not be related to bundle delete. Also, the session errors in the log don't seem to map directly to this operation. Those are generated when the gui session is inactive for too long.

I have successfully deleted a large bundle, 104MB, multiple times now and can't reproduce.

I have not tried an EAP5 bundle, that will be the next test, It's almost 3 times the size but delete, in general, does not perform streaming or caching of the
bundle files.

Please verify again that you think this is specific to bundle delete and that the OOM is directly related.

Also, can you successfully delete smaller bundles?  And what is the memory allocated to the server JVM?

Currently I'm considering this "unable to reproduce".
Comment 5 Sudhir D 2010-07-09 04:23:37 EDT
I tested with eap 5 bundle (~300MB) and was able to delete successfully. I'm using jon-server-2.4.0.GA_QA Build# 53.

However, once deleted, the bundle name still shows up and page does not get refreshed unless we leave the GWT session and come back again. For this I will raise another bug. 

Marking this one as verified.
Comment 6 Jay Shaughnessy 2010-07-15 16:58:01 EDT
I'm resetting this to ON_DEV because I think if you use a large enough bundle file you can reproduce this. It seems perhaps that the PackageBits for a BundleFile's PackageVersion are getting eager loaded when they should not be.
Comment 7 Charles Crouch 2010-07-19 16:02:46 EDT
Dropping priority, since this won't be ready for the release

(2:46:32 PM) jshaughn: ccrouch:  ok, I have it working.  I'm doing a lot of cleanup at this point.  It will still take me the rest of the day to fully test.
(2:46:57 PM) jshaughn: I have to validate again after the cleanup, and then do oracle.
(2:47:24 PM) jshaughn: it's not a no-op as it does touch a variety of content code.
(2:47:38 PM) jshaughn: also have to probably fix some broken tests
(2:48:12 PM) jshaughn: probablt ready for review in the am.
(2:49:20 PM) jshaughn: if I commit it we'll need to retest bundle stuff, large bundle stuff, and any other content subsystem stuff.
(2:49:38 PM) jshaughn: like content sources
(2:52:31 PM) ccrouch: jshaughn: ok, thanks for the summary
(2:52:50 PM) ccrouch: i think at this stage, that is not something we want to try to consume in the release branch
(2:53:15 PM) jshaughn: i figured
Comment 8 Jay Shaughnessy 2010-07-20 09:55:46 EDT
The problem here turned out to be some subtlety in the way Hibernate performs lazy load.  Basically, the Lob field in PackageBits is (by default using @Lob) set for lazy fetch. But, Hibernate by default does not honor lazy load at the property (field) level, only on mapped relations.  In this case the Cascade Removal of a BundleFile/PackageVersion/PackageBits triggered the load of PackageBits.bits field. Given its size it generates the OOM. Note that this is not specific to delete, any fetch of the PackageBits would do this. Also, it's not specific to Bundles, it's true for any manipulation of large PackageVersion content. 

Hibernate supports instrumenting the entity class (at compile time), basically adding additional support in the class to perform the extra logic of property level lazy fetch. I was able to implement this and it did solve the issue. But, unfortunately the class instrumentation adds Hibernate class dependencies to the instrumented class. This violates our restriction on requiring Hibernate classes in the Agent or Remote Client.

The alternative suggested on the web is to force Hibernate to apply its standard lazy load on relational mappings.  To do this you extract out the (Lob) field that requires lazy loading into a new entity and create a required 1-1 mapping.  This is what has now been implemented. 

PackageBitsBlob is a new entity that is now the actual Lob contents for a PackageBits entity.  Note that PackageBits and PackageBitsBlob are two entities that *share* the same db row.  PackageBits declares a required 1-1 mapping to PackageBitsBlob, allowing Lazy load semantics on the Lob.  Care must be taken here to ensure that constraints are not violated when setting this up.

Related Links on this topic:

http://docs.jboss.org/hibernate/stable/core/reference/en/html/performance.html#performance-fetching-lazyproperties

http://community.jboss.org/wiki/Someexplanationsonlazyloadingone-to-one

http://docs.jboss.org/hibernate/stable/annotations/reference/en/html_single/#entity-hibspec-singleassoc

http://community.jboss.org/wiki/AShortPrimerOnFetchingStrategies

http://docs.codehaus.org/display/MAVENUSER/Howto+instrument+domain+model+classes+using+hibernate
Comment 9 Jay Shaughnessy 2010-07-20 12:15:19 EDT
fix commit: 67fe77e638ef6bfb394b81476eaefc2830364199

QA Note: This is not in release-3.0.0. It is in master/4.0.0


For anyone hitting this in RHQ 3.0.0/JON 2.4 the workarounds are:

1) Increase memory to the RHQ server JVM to compensate for the OOM.  

2) Delete the offending rhq_package_bits row manually from the DB and try again. You'll probably need to set rhq_package_version.package_bits_id to null first to break the FK relationship.
Comment 10 Sudhir D 2010-10-15 07:27:57 EDT
Verified this on rhq-server-4.0.0-SNAPSHOT build# 415. I was able to successfully upload and delete bundle. 

Marking this verified.
Comment 11 Heiko W. Rupp 2013-09-02 03:26:52 EDT
Bulk closing of issues that were VERIFIED, had no target release and where the status changed more than a year ago.

Note You need to log in before you can comment on or make changes to this bug.