Created attachment 429506 [details] server log Description of problem: Delete bundle is not working. It gives exceptions in server log. 2010-07-05 16:12:28,832 INFO [STDOUT] SerialUtility.prepare [findBundleVersionsByCriteria] Detached in: 1ms, Size is: 5977 2010-07-05 16:12:48,065 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/coregui].[org.rhq.enterprise.gui.coregui.CoreGUI AlertGWTService]] Servlet.service() for servlet org.rhq.enterprise.gui.coregui.CoreGUI AlertGWTService threw exception java.lang.RuntimeException: Failed to validate session at org.rhq.enterprise.gui.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:55) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:619) Caused by: org.rhq.enterprise.server.auth.SessionNotFoundException at org.rhq.enterprise.server.auth.SessionManager.getSubject(SessionManager.java:167) at org.rhq.enterprise.server.auth.SubjectManagerBean.getSubjectBySessionId(SubjectManagerBean.java:568) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77) at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156) at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy245.getSubjectBySessionId(Unknown Source) at org.rhq.enterprise.gui.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:53) ... 22 more Version-Release number of selected component (if applicable): JON 2.4 GA_QA (tag-jon-release build# 49) build number: 10775:e086474 How reproducible: Always Steps to Reproduce: 1. Go to Administration->Content->Bundles 2. Click on button 'New'. 3. Select the radio button 'Upload'. 4. Click on the 'Browse' button. 5. Browse and select the archive file. (provide valid archive file) 6. Click on Next button and proceed for bundle creation. (complete bundle creation) 7. Select bundle and click on 'Delete' button. Actual results: Delete bundle is not working. It gives exceptions in server log. Expected results: Bundle should delete without exception in server log. Additional info: Please refer attached server log.
Bundle deletion gives error: Title : Failed to delete bundle [example] Severity : Error Time : Tue 06 Jul 2010 04:41:40 PM IST Detail : java.lang.RuntimeException: javax.ejb.EJBTransactionRolledbackException:java.lang.OutOfMemoryError: Java heap space -> java.lang.RuntimeException:java.lang.OutOfMemoryError: Java heap space -> java.lang.OutOfMemoryError:Java heap space
Rajan, can you provide more details as to the configuration you tested? How large was the bundle you attempted to deploy? Dev to open separate bug investigating possible size limitations/recommendations.
Test environment: Platform: RHEL 5.5 Database: Postgres 8.4 Java: Sun JDK1.6 Browser: FF 3.0.19 The bundle size was 294.0MB and it contains jboss-eap-5.0.0.GA.zip
The error reported here, OutOfMemory, may not be related to bundle delete. Also, the session errors in the log don't seem to map directly to this operation. Those are generated when the gui session is inactive for too long. I have successfully deleted a large bundle, 104MB, multiple times now and can't reproduce. I have not tried an EAP5 bundle, that will be the next test, It's almost 3 times the size but delete, in general, does not perform streaming or caching of the bundle files. Please verify again that you think this is specific to bundle delete and that the OOM is directly related. Also, can you successfully delete smaller bundles? And what is the memory allocated to the server JVM? Currently I'm considering this "unable to reproduce".
I tested with eap 5 bundle (~300MB) and was able to delete successfully. I'm using jon-server-2.4.0.GA_QA Build# 53. However, once deleted, the bundle name still shows up and page does not get refreshed unless we leave the GWT session and come back again. For this I will raise another bug. Marking this one as verified.
I'm resetting this to ON_DEV because I think if you use a large enough bundle file you can reproduce this. It seems perhaps that the PackageBits for a BundleFile's PackageVersion are getting eager loaded when they should not be.
Dropping priority, since this won't be ready for the release (2:46:32 PM) jshaughn: ccrouch: ok, I have it working. I'm doing a lot of cleanup at this point. It will still take me the rest of the day to fully test. (2:46:57 PM) jshaughn: I have to validate again after the cleanup, and then do oracle. (2:47:24 PM) jshaughn: it's not a no-op as it does touch a variety of content code. (2:47:38 PM) jshaughn: also have to probably fix some broken tests (2:48:12 PM) jshaughn: probablt ready for review in the am. (2:49:20 PM) jshaughn: if I commit it we'll need to retest bundle stuff, large bundle stuff, and any other content subsystem stuff. (2:49:38 PM) jshaughn: like content sources (2:52:31 PM) ccrouch: jshaughn: ok, thanks for the summary (2:52:50 PM) ccrouch: i think at this stage, that is not something we want to try to consume in the release branch (2:53:15 PM) jshaughn: i figured
The problem here turned out to be some subtlety in the way Hibernate performs lazy load. Basically, the Lob field in PackageBits is (by default using @Lob) set for lazy fetch. But, Hibernate by default does not honor lazy load at the property (field) level, only on mapped relations. In this case the Cascade Removal of a BundleFile/PackageVersion/PackageBits triggered the load of PackageBits.bits field. Given its size it generates the OOM. Note that this is not specific to delete, any fetch of the PackageBits would do this. Also, it's not specific to Bundles, it's true for any manipulation of large PackageVersion content. Hibernate supports instrumenting the entity class (at compile time), basically adding additional support in the class to perform the extra logic of property level lazy fetch. I was able to implement this and it did solve the issue. But, unfortunately the class instrumentation adds Hibernate class dependencies to the instrumented class. This violates our restriction on requiring Hibernate classes in the Agent or Remote Client. The alternative suggested on the web is to force Hibernate to apply its standard lazy load on relational mappings. To do this you extract out the (Lob) field that requires lazy loading into a new entity and create a required 1-1 mapping. This is what has now been implemented. PackageBitsBlob is a new entity that is now the actual Lob contents for a PackageBits entity. Note that PackageBits and PackageBitsBlob are two entities that *share* the same db row. PackageBits declares a required 1-1 mapping to PackageBitsBlob, allowing Lazy load semantics on the Lob. Care must be taken here to ensure that constraints are not violated when setting this up. Related Links on this topic: http://docs.jboss.org/hibernate/stable/core/reference/en/html/performance.html#performance-fetching-lazyproperties http://community.jboss.org/wiki/Someexplanationsonlazyloadingone-to-one http://docs.jboss.org/hibernate/stable/annotations/reference/en/html_single/#entity-hibspec-singleassoc http://community.jboss.org/wiki/AShortPrimerOnFetchingStrategies http://docs.codehaus.org/display/MAVENUSER/Howto+instrument+domain+model+classes+using+hibernate
fix commit: 67fe77e638ef6bfb394b81476eaefc2830364199 QA Note: This is not in release-3.0.0. It is in master/4.0.0 For anyone hitting this in RHQ 3.0.0/JON 2.4 the workarounds are: 1) Increase memory to the RHQ server JVM to compensate for the OOM. 2) Delete the offending rhq_package_bits row manually from the DB and try again. You'll probably need to set rhq_package_version.package_bits_id to null first to break the FK relationship.
Verified this on rhq-server-4.0.0-SNAPSHOT build# 415. I was able to successfully upload and delete bundle. Marking this verified.
Bulk closing of issues that were VERIFIED, had no target release and where the status changed more than a year ago.