Bug 702390

Summary: Large number of BundleDeployments for a single Bundle causes fetch serialization issues
Product: [Other] RHQ Project Reporter: Jay Shaughnessy <jshaughn>
Component: Core ServerAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: medium    
Version: 3.0.0CC: hrupp, loleary, mazz, skondkar
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: 4.0.1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-02 07:20:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 678340, 702748    

Description Jay Shaughnessy 2011-05-05 14:32:45 UTC
Bundles with a lot of deployments, roughly >=50, will cause the HibernateDetachUtility to throw exceptions like:

2011-05-02 12:54:45,534 WARN  [org.rhq.enterprise.server.util.HibernateDetachUtility] Getting different object hierarchies back from calls: org.rhq.core.domain.bundle.BundleDeployment

Although these are warnings, and the detach utility "scrubbing" completes, subsequent serialization of the data may fail with errors like:

Caused by: org.hibernate.LazyInitializationException: could not initialize proxy - no Session

This can then cause issues in the remote client, like CoreGUI, which can have unpredictable results.  CoreGUI has been reported to go into a "loop" when tying to render the deployments for the bundle.

Comment 1 Jay Shaughnessy 2011-05-05 14:40:52 UTC
For a Bundle there is a "Live" BundleDeployment and it replaces the previous live deployment.  The exception being the first deployment which does not replace anything.

The BundleDeployment entity defines a OneToOne relationship to itself to define the 'replacedBundleDeployment' field.  This is eager fetched, meaning that when fetching the live deployment hibernate has no choice but to also fetch the replacedBundleDeployment.  This then happens recursively, grabbing the entire history of BundleDeployments.

The HibernateDetachUtility has a max depth to prevent against unexpected recursion issues, and this is what is getting violated (the WARNING in the description text).

This can not be lazy fetched (we already mistakenly tried to do this, resulting in the problem) because the association is optional.  Hibernate must eager fetch optional XxxToOne associations because it must check for null. 

[Note - we should go through our domain model and look for this mistake in other places]

So, the proposed solution is to remove the entity association and treat the field as an id Integer field, thus preventing any [recursive] entity fetch.

Comment 2 Jay Shaughnessy 2011-05-05 18:01:23 UTC
commit 976c5fc6ed0e7e5354ea6a890caafa58b3f1c7bc
Author: Jay Shaughnessy <jshaughn>
Date:   Thu May 5 13:45:09 2011 -0400

Remove self-referring entity association for BundleDeployment.replacedBundleDeployment. This prevents eager loading of the entire deployment chain when fetching a BundleDeployment, and in turn avoids issues with Hibernate class scrubbing when remote fetching deployments.

Instead of using the entity as the field member, just use the ID.


Test Notes:
Please test this by exercising bundle create/deploy/revert. As well as delete of version and destination with deployments.

Finally, with at least 5 deployments for a single destination tune down the following setting (see bug 602109) :

rhq.server.hibernate-detach-utility.depth-allowed=5

Then ensure that you can still successfully view the depoyment list for the destination, in the gui.

Comment 3 Charles Crouch 2011-05-09 20:05:34 UTC
Adding for triage for RHQ 4.0.1 release

Comment 4 Jay Shaughnessy 2011-05-10 22:04:57 UTC
Cherry Picked the following three commits from master to release-4.0.0 to support this:

4e75aadafeb841c6e4c8030265197ec029efb28a
2dcc812cb2e84ccff01dc62824958baded28eec6
79fe9be4b6ef8adf0c376221438afacab1c2014c

Comment 5 Sunil Kondkar 2011-05-20 12:24:43 UTC
Verified on build#38 (Version: 4.0.1-SNAPSHOT Build Number: a104cdf)

Verified bundle creation, deploy and revert. Deleted versions and destinations with deployments.

Deployed a bundle 5 times at a single destination and had an entry in rhq-server.properties.

I was able to view the deployment list for the destination, in the gui.

Marking as verified.

Comment 6 Heiko W. Rupp 2013-09-02 07:20:36 UTC
Bulk closing of issues that were VERIFIED, had no target release and where the status changed more than a year ago.