Bug 1380158

Looks to possibly related to Rails 5.  Here is where it seems to be blowing up based on the stack trace:

From /var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_engine.rb
====================================

  def self.deliver(*args)
    # This was added because in RAILS 5 PUMA sends multiple requests
    # to the same process and our DRb server implementation for Automate
    # Methods is not thread safe. The permit_concurrent_loads allows the
    # DRb requests which run in different threads to load constants.
    ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
      @deliver_mutex.synchronize { deliver_block(*args) }
    end
  end

Madhu - Believe this is happening on 5.6.1.  Would this be fixed by PR https://github.com/ManageIQ/manageiq/pull/10445 ?

Created attachment 1205680 [details]
Proposed Patch

Please make a back up of 
lib/miq_automation_engine/engine/miq_ae_engine.rb
lib/miq_automation_engine/engine/miq_ae_method.rb
lib/miq_automation_engine/engine/miq_ae_workspace.rb

And copy over the files from the tar which has one new file
lib/miq_automation_engine/engine/drb_remote_invoker.rb
in addition to the updates to the above 3

Josh,
Can you please test it with the enclosed patch.
Thanks,
Madhu

Josh,
Can you please test it with the enclosed patch.
Thanks,
Madhu

Fixed in PR https://github.com/ManageIQ/manageiq/pull/10135

Verification steps please

This issue started happening with Rails 5, where the multi threaded PUMA server was introduced. This PUMA server can send multiple requests into the same process, which ends up running multiple threads to handle the requests. When multiple requests are running the automate method which use the module level DRb methods were stepping on each other. So what would happen is that 2 threads running within the same process will step on the DRb variables. 
So the code was changed to use the non DRb module methods.

To test it

 You would need to define a custom button with multiple Dynamic Dialogs
 Since each Dynamic Dialog Method would have a method you can introduce a slight delay in the methods sleep(30).
 So you would click on 1 dynamic field which will say fetch a list of names
 While that is running click on the 2nd dynamic field which will fetch a list of some other names. This method should not have a delay (sleep)

Then you can check the logs to see that multiple requests came in 

3381 [----] I, [2016-11-08T17:59:57.107948 #17105:3fecb845e1d0]  INFO -- : MIQ(MiqQueue#delivered) M     essage id: [32590], State: [error], Delivered in [7.510469] seconds

In the above line #17105 is the process id and  3fecb845e1d0 is the process id.
so in this case we would be seeing multiple treads running in parallel with the same process id and the log lines would be interleaved. While the first method is waiting for 30 seconds the second would come in on a different thread and it would end before the first one ends.

Yes dynamic fields, where you would have multiple independent dynamic fields in a dialog.

I think you need to create a method that returns a list of items that you can use in your dynamic field.
A sample example looks like this https://github.com/ManageIQ/manageiq/blob/master/db/fixtures/ae_datastore/ManageIQ/Cloud/Orchestration/Operations/Methods.class/__methods__/available_flavors.rb


This is a sample method to get a static list of values
Which can be used in a dropdown

#
# Sample method with static data
#

capital_list = { 'USA' => 'Washington DC', 'Russia' => 'Moscow', 'Japan' => 'Tokyo', 'France' => 'Paris', 'Canada' => 'Ottawa' }

capital_list[nil] = "<Choose>"

dialog_field = $evm.object

# sort_by: value / description / none
dialog_field["sort_by"] = "description"

# sort_order: ascending / descending
dialog_field["sort_order"] = "ascending"

# data_type: string / integer
dialog_field["data_type"] = "string"

# required: true / false
dialog_field["required"] = "true"

dialog_field["values"] = capital_list
dialog_field["default_value"] = nil



You can then wire that method to a drop down in a dynamic dialog.

If you need to setup a dynamic dialog please read https://access.redhat.com/mastering-cloudforms-automation

Chapter 29 on Service Dialogs talks about how to integrate with dynamic fields.

Verified on 5.7.0.11-rc1.20161115160629_46cf4f1. I followed the steps which were described in comment10. I created two methods and the first one contained sleep(10). In automation.log I found that these two methods executed in the separate threads and second one finished before the first one.
[----] I, [2016-11-21T10:57:57.760882 #24380:12e6818] INFO -- : <AEMethod [/Test/System/Request/Test1]> Starting
[----] I, [2016-11-21T10:57:59.758649 #24380:12e9d88] INFO -- : <AEMethod [/Test/System/Request/Test2]> Starting
[----] I, [2016-11-21T10:58:00.736631 #24380:12e9d88] INFO -- : <AEMethod [/Test/System/Request/Test2]> Ending
[----] I, [2016-11-21T10:58:09.213614 #24380:12e6818] INFO -- : <AEMethod [/Test/System/Request/Test1]> Ending

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0012.html