Looks to possibly related to Rails 5. Here is where it seems to be blowing up based on the stack trace: From /var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_engine.rb ==================================== def self.deliver(*args) # This was added because in RAILS 5 PUMA sends multiple requests # to the same process and our DRb server implementation for Automate # Methods is not thread safe. The permit_concurrent_loads allows the # DRb requests which run in different threads to load constants. ActiveSupport::Dependencies.interlock.permit_concurrent_loads do @deliver_mutex.synchronize { deliver_block(*args) } end end
Madhu - Believe this is happening on 5.6.1. Would this be fixed by PR https://github.com/ManageIQ/manageiq/pull/10445 ?
Created attachment 1205680 [details] Proposed Patch Please make a back up of lib/miq_automation_engine/engine/miq_ae_engine.rb lib/miq_automation_engine/engine/miq_ae_method.rb lib/miq_automation_engine/engine/miq_ae_workspace.rb And copy over the files from the tar which has one new file lib/miq_automation_engine/engine/drb_remote_invoker.rb in addition to the updates to the above 3
Josh, Can you please test it with the enclosed patch. Thanks, Madhu
Fixed in PR https://github.com/ManageIQ/manageiq/pull/10135
Verification steps please
This issue started happening with Rails 5, where the multi threaded PUMA server was introduced. This PUMA server can send multiple requests into the same process, which ends up running multiple threads to handle the requests. When multiple requests are running the automate method which use the module level DRb methods were stepping on each other. So what would happen is that 2 threads running within the same process will step on the DRb variables. So the code was changed to use the non DRb module methods. To test it You would need to define a custom button with multiple Dynamic Dialogs Since each Dynamic Dialog Method would have a method you can introduce a slight delay in the methods sleep(30). So you would click on 1 dynamic field which will say fetch a list of names While that is running click on the 2nd dynamic field which will fetch a list of some other names. This method should not have a delay (sleep) Then you can check the logs to see that multiple requests came in 3381 [----] I, [2016-11-08T17:59:57.107948 #17105:3fecb845e1d0] INFO -- : MIQ(MiqQueue#delivered) M essage id: [32590], State: [error], Delivered in [7.510469] seconds In the above line #17105 is the process id and 3fecb845e1d0 is the process id. so in this case we would be seeing multiple treads running in parallel with the same process id and the log lines would be interleaved. While the first method is waiting for 30 seconds the second would come in on a different thread and it would end before the first one ends.
Yes dynamic fields, where you would have multiple independent dynamic fields in a dialog.
I think you need to create a method that returns a list of items that you can use in your dynamic field. A sample example looks like this https://github.com/ManageIQ/manageiq/blob/master/db/fixtures/ae_datastore/ManageIQ/Cloud/Orchestration/Operations/Methods.class/__methods__/available_flavors.rb This is a sample method to get a static list of values Which can be used in a dropdown # # Sample method with static data # capital_list = { 'USA' => 'Washington DC', 'Russia' => 'Moscow', 'Japan' => 'Tokyo', 'France' => 'Paris', 'Canada' => 'Ottawa' } capital_list[nil] = "<Choose>" dialog_field = $evm.object # sort_by: value / description / none dialog_field["sort_by"] = "description" # sort_order: ascending / descending dialog_field["sort_order"] = "ascending" # data_type: string / integer dialog_field["data_type"] = "string" # required: true / false dialog_field["required"] = "true" dialog_field["values"] = capital_list dialog_field["default_value"] = nil You can then wire that method to a drop down in a dynamic dialog. If you need to setup a dynamic dialog please read https://access.redhat.com/mastering-cloudforms-automation
Chapter 29 on Service Dialogs talks about how to integrate with dynamic fields.
Verified on 5.7.0.11-rc1.20161115160629_46cf4f1. I followed the steps which were described in comment10. I created two methods and the first one contained sleep(10). In automation.log I found that these two methods executed in the separate threads and second one finished before the first one. [----] I, [2016-11-21T10:57:57.760882 #24380:12e6818] INFO -- : <AEMethod [/Test/System/Request/Test1]> Starting [----] I, [2016-11-21T10:57:59.758649 #24380:12e9d88] INFO -- : <AEMethod [/Test/System/Request/Test2]> Starting [----] I, [2016-11-21T10:58:00.736631 #24380:12e9d88] INFO -- : <AEMethod [/Test/System/Request/Test2]> Ending [----] I, [2016-11-21T10:58:09.213614 #24380:12e6818] INFO -- : <AEMethod [/Test/System/Request/Test1]> Ending
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0012.html