Red Hat Bugzilla – Bug 1284573
Intermittent "RangeError: 0xnnnnnnn is recycled object" errors when running automation scripts
Last modified: 2016-02-10 10:20:14 EST
Created attachment 1097724 [details] Script to reproduce the problem Description of problem: We get occasional "RangeError: 0xnnnnnnn is recycled object" errors when running any of several automation scripts. The error seems to be related to stale association references, i.e. ems = $evm.vmdb('ems').find(:first) hosts = ems.hosts # at some stage later... hosts.each do |host| Version-Release number of selected component (if applicable): 5.4.3.1 How reproducible: The attached script can reproduce the problem for me around 1 in 5 times of running it. Each of the loops exercises the automation engine more or less. With: loops = [3,4] I get the recycled object error fairly frequently. If you add in the other loops: loops = [1,2,3,4] the execution takes much longer, and other symptoms of failure occur, generally looking like memory corruption/re-use, i.e. [----] I, [2015-11-19T14:39:29.055237 #2692:a81f204] INFO -- : <AEMethod test_pemcg> First loop... [----] I, [2015-11-19T14:41:57.197517 #2692:a81f204] INFO -- : <AEMethod test_pemcg> Second loop... [----] I, [2015-11-19T14:43:33.635100 #2692:a81f204] INFO -- : <AEMethod test_pemcg> Third loop... [----] I, [2015-11-19T14:43:33.635827 #2692:a81f204] INFO -- : <AEMethod test_pemcg> Fourth loop... [----] E, [2015-11-19T14:43:33.637500 #2692:a85d798] ERROR -- : <AEMethod test_pemcg> The following error occurred during method evaluation: [----] E, [2015-11-19T14:43:33.637865 #2692:a85d798] ERROR -- : <AEMethod test_pemcg> NoMethodError: undefined method `each' for "2015-11-10 15:59:32.79544":String [----] E, [2015-11-19T14:43:33.638485 #2692:a85d798] ERROR -- : <AEMethod test_pemcg> (druby://127.0.0.1:41919) /opt/rh/ruby200/root/usr/share/ruby/drb/invokemethod.rb:14:in `perform_with_block' Steps to Reproduce: 1. Run the attached script through simulation 2. 3. Actual results: Expected results: Additional info:
https://github.com/ManageIQ/manageiq/pull/5590
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/5e05ff17373f0cc9c2bfd5c81eb636dbfad9f250 commit 5e05ff17373f0cc9c2bfd5c81eb636dbfad9f250 Author: Madhu Kanoor <mkanoor@redhat.com> AuthorDate: Tue Nov 24 16:04:19 2015 -0500 Commit: Madhu Kanoor <mkanoor@redhat.com> CommitDate: Tue Nov 24 17:29:29 2015 -0500 Array object was not being cached https://bugzilla.redhat.com/show_bug.cgi?id=1284573 The Array object that is sent back to the Automate Method via Drb is not being cached, leading to intermittent Recycled Object error The individual objects in the Array were cached but not the Array itself. lib/miq_automation_engine/engine/miq_ae_service_model_base.rb | 4 ++-- spec/lib/miq_automation_engine/miq_ae_service_spec.rb | 9 +++++++++ 2 files changed, 11 insertions(+), 2 deletions(-)
Could we have this back-ported to CFME 5.4 please?
5.4.z bugzilla is https://bugzilla.redhat.com/show_bug.cgi?id=1286864
Merge Request for 5.5.z http://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/merge_requests/536
New commit detected on cfme/5.4.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=3a195ee826c4aa84555a9c809b5a8d81c662cc52 commit 3a195ee826c4aa84555a9c809b5a8d81c662cc52 Author: Madhu Kanoor <mkanoor@redhat.com> AuthorDate: Mon Nov 30 11:01:52 2015 -0500 Commit: Madhu Kanoor <mkanoor@redhat.com> CommitDate: Mon Nov 30 11:01:52 2015 -0500 Array object was not being cached, Drb Recycled Object error https://bugzilla.redhat.com/show_bug.cgi?id=1284573 The Array object that is sent back to the Automate Method via Drb is not being cached, leading to intermittent Recycled Object error The individual objects in the Array were cached but not the Array itself. .../engine/miq_ae_service_model_base.rb | 2 +- .../spec/lib/miq_automation_engine/miq_ae_service_spec.rb | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-)
New commit detected on cfme/5.4.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=c52a301636783c97acea0a3353471e1bb51866ac commit c52a301636783c97acea0a3353471e1bb51866ac Merge: 89a28ce 3a195ee Author: Greg McCullough <gmccullo@redhat.com> AuthorDate: Mon Nov 30 17:42:47 2015 -0500 Commit: Greg McCullough <gmccullo@redhat.com> CommitDate: Mon Nov 30 17:42:47 2015 -0500 Merge branch 'bugzilla_1284573_5_4_z' into '5.4.z' Array object was not being cached, Drb Recycled Object error https://bugzilla.redhat.com/show_bug.cgi?id=1284573 The Array object that is sent back to the Automate Method via Drb is not being cached, leading to intermittent Recycled Object error The individual objects in the Array were cached but not the Array itself. Original PR https://github.com/ManageIQ/manageiq/pull/5590 Manual changes made to the 2 files. See merge request !554 .../engine/miq_ae_service_model_base.rb | 2 +- .../spec/lib/miq_automation_engine/miq_ae_service_spec.rb | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-)
Waiting on MR http://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/merge_requests/536 to get merged
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=d1694299ff35428d9117d309d7d637b90ece5d68 commit d1694299ff35428d9117d309d7d637b90ece5d68 Author: Madhu Kanoor <mkanoor@redhat.com> AuthorDate: Tue Nov 24 16:04:19 2015 -0500 Commit: Madhu Kanoor <mkanoor@redhat.com> CommitDate: Wed Nov 25 09:24:57 2015 -0500 Array object was not being cached https://bugzilla.redhat.com/show_bug.cgi?id=1284573 The Array object that is sent back to the Automate Method via Drb is not being cached, leading to intermittent Recycled Object error The individual objects in the Array were cached but not the Array itself. lib/miq_automation_engine/engine/miq_ae_service_model_base.rb | 4 ++-- spec/lib/miq_automation_engine/miq_ae_service_spec.rb | 9 +++++++++ 2 files changed, 11 insertions(+), 2 deletions(-)
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=c2fd5e17c30d16f416a35fc485a7cf5a444af63f commit c2fd5e17c30d16f416a35fc485a7cf5a444af63f Merge: 3a76459 d169429 Author: Greg McCullough <gmccullo@redhat.com> AuthorDate: Mon Dec 7 13:44:49 2015 -0500 Commit: Greg McCullough <gmccullo@redhat.com> CommitDate: Mon Dec 7 13:44:49 2015 -0500 Merge branch 'bugzilla_1284573_5_5_z' into '5.5.z' Array object was not being cached https://bugzilla.redhat.com/show_bug.cgi?id=1284573 The Array object that is sent back to the Automate Method via Drb is not being cached, leading to intermittent Recycled Object error The individual objects in the Array were cached but not the Array itself. Clean Cherry Pick from Original PR https://github.com/ManageIQ/manageiq/pull/5590 See merge request !536 lib/miq_automation_engine/engine/miq_ae_service_model_base.rb | 4 ++-- spec/lib/miq_automation_engine/miq_ae_service_spec.rb | 9 +++++++++ 2 files changed, 11 insertions(+), 2 deletions(-)
Verified fixed in 5.2.2.1 - 5.5.2.1.20160114044944_395c086. Tested multiple times with 'loops' set to both [3,4] and [1,2,3,4] and I wasn't able to reproduce the issue anymore.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:0159