Bug 1297952

Summary: CF 3.2: Automation code throwing "0x000... is recycled object" exceptions
Product: Red Hat CloudForms Management Engine Reporter: Chris Pelland <cpelland>
Component: AutomateAssignee: mkanoor
Status: CLOSED ERRATA QA Contact: Milan Falešník <mfalesni>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 5.4.0CC: cpelland, dajohnso, gmccullo, jhardy, jocarter, jprause, mfalesni, mfeifer, mkanoor, nachandr, obarenbo, simaishi, ssainkar, tfitzger
Target Milestone: GAKeywords: ZStream
Target Release: 5.4.5   
Hardware: All   
OS: All   
Whiteboard: automate
Fixed In Version: 5.4.5.2 Doc Type: Bug Fix
Doc Text:
After the CloudForms 3.2 migration, automation code threw the following exception when executing a best_fit_code method for bulk provision testing: “0x000... is recycled object” This was because previous version of CloudForms used its own caching mechanism to cache objects sent as references to the Automate Methods. The DRb recycled object errors were caused since some of the objects which were sent as references were not being cached. DRb has a built in caching mechanism based on TimerIdConv, which is now being implemented with this patch. This, as a result, has now resolved the issue.
Story Points: ---
Clone Of: 1297097 Environment:
Last Closed: 2016-03-29 14:16:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1297097    
Bug Blocks: 1297499    
Attachments:
Description Flags
Fix none

Comment 1 mkanoor 2016-01-19 19:46:40 UTC
Created attachment 1116346 [details]
Fix

Please replace the enclosed 2 files into the vmdb directory

Comment 2 mkanoor 2016-02-04 14:58:23 UTC
Merge Request
http://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/merge_requests/731

Pending to be merged

Comment 3 CFME Bot 2016-02-09 22:48:00 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a6890159511fda0773ed8dcd17402117df6c89f9

commit a6890159511fda0773ed8dcd17402117df6c89f9
Merge: 40a99c4 0945d50
Author:     Greg McCullough <gmccullo>
AuthorDate: Tue Feb 9 17:30:01 2016 -0500
Commit:     Greg McCullough <gmccullo>
CommitDate: Tue Feb 9 17:30:01 2016 -0500

    Merge branch 'bugzilla_1297952_5_4_z' into '5.4.z'
    
    Use DRb's builtin caching mechanism
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1297952
    
    Previously we have been using our own caching mechanism to
    cache objects sent as references to the Automate Methods. We have
    gotten DRb recycled object errors, since some of the objects which
    were sent as references were not being cached.
    DRb has a builtin caching mechanism based on TimerIdConv, which
    we are starting to use in this PR.
    
    Original PR
    https://github.com/ManageIQ/manageiq/pull/6190
    
    Not a clean cherry-pick, the spec file had conflict when applying the patch because of old vmdb directory in 5.4
    
    See merge request !731

 vmdb/lib/miq_automation_engine/engine/miq_ae_method.rb    |  8 ++++++++
 vmdb/lib/miq_automation_engine/engine/miq_ae_service.rb   |  2 +-
 .../spec/lib/miq_automation_engine/miq_ae_service_spec.rb | 15 ---------------
 3 files changed, 9 insertions(+), 16 deletions(-)

Comment 6 mkanoor 2016-03-23 13:40:40 UTC
This is very difficult to recreate, because it depends on when the Garbage Collector ran on the Engine side and free'ed the object for which the automate method is still holding on to the handle and tries to access it. I think we should just test the functionality and make sure it doesn't break.

Comment 7 Milan Falešník 2016-03-23 17:06:33 UTC
Checked in 5.4.5.2. After discussion with mkanoor I used the script from [1] which I ran about a dozen times and no error ever happened. The provisioning also works so ...

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1284573

Comment 9 errata-xmlrpc 2016-03-29 14:16:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0171.html