Bug 1413629 - ae_result = retry causes DRb::DRbConnError inside of iteration of VMDB objects
Summary: ae_result = retry causes DRb::DRbConnError inside of iteration of VMDB objects
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Automate
Version: 5.6.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: GA
: cfme-future
Assignee: mkanoor
QA Contact: Dave Johnson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-16 14:59 UTC by Jeffrey Cutter
Modified: 2019-08-06 20:06 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-20 11:55:03 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)

Description Jeffrey Cutter 2017-01-16 14:59:07 UTC
Description of problem:

Issuing an ae_result = retry and exiting inside of an iteration leads to a DRb::DRbConnError.  Moving the retry / exit MIQ_OK outside of the iteration is successful.

How reproducible:

Code that causes error:

  service.vms.each do |vm|
    if vm.ipaddresses.first.blank?
      $evm.log(:info, "IP Address not yet available for #{vm.name} wait 30 seconds.")
      $evm.root['ae_result'] = 'retry'
      $evm.root['ae_retry_interval'] = '30.seconds'
      exit MIQ_OK
    end
  end

Code that works:

  need_retry = false
 
  service.vms.each do |vm|
    if vm.ipaddresses.first.blank?
      $evm.log(:info, "IP Address not yet available for #{vm.name}")
      need_retry = true
    end
  end

  if need_retry
    $evm.log(:info, "Scheduling retry in 30 seconds to wait for IP addresses")
    $evm.root['ae_result'] = 'retry'
    $evm.root['ae_retry_interval'] = '30.seconds'
    exit MIQ_OK
  end

Additional info:

Error message encountered:

[----] I, [2017-01-11T14:44:02.996107 #3756:60f838]  INFO -- : Q-task_id([service_template_provision_task_1000000000412]) <AEMethod createcsv> IP Address not yet available for lab-slgx0546-0, wait 30 seconds
[----] I, [2017-01-11T14:44:02.997219 #3756:60f838]  INFO -- : Q-task_id([service_template_provision_task_1000000000412]) <AEMethod createcsv> retry set
[----] I, [2017-01-11T14:44:02.998308 #3756:60f838]  INFO -- : Q-task_id([service_template_provision_task_1000000000412]) <AEMethod createcsv> retry interval set
[----] E, [2017-01-11T14:44:03.000961 #3756:60f838] ERROR -- : Q-task_id([service_template_provision_task_1000000000412]) <AEMethod createcsv> [(DRb::DRbConnError)connection closed]
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:578:in `load'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:640:in `recv_reply'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:940:in `recv_reply'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1254:in `send_message'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1142:in `block (2 levels) in method_missing'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1229:in `open'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1141:in `block in method_missing'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1160:in `with_friend'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1140:in `method_missing'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/invokemethod.rb:10:in `block_yield'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/invokemethod.rb:17:in `block in perform_with_block'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/invokemethod.rb:14:in `each'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/invokemethod.rb:14:in `perform_with_block'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1582:in `perform'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1657:in `block (2 levels) in main_loop'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1653:in `loop'
(druby://127.0.0.1:34951) /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1653:in `block in main_loop'
(druby://127.0.0.1:34951) /opt/rh/cfme-gemset/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
(druby://127.0.0.1:34951) /opt/rh/cfme-gemset/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
<code: service.vms.each do |vm|>:77:in `<main>'
[----] I, [2017-01-11T14:44:03.036892 #3756:51f98c]  INFO -- : Q-task_id([service_template_provision_task_1000000000412]) <AEMethod [/Desjardins/Cloud/Orchestration/Provisioning/StateMachines/Methods/createcsv]> Ending
[----] I, [2017-01-11T14:44:03.037233 #3756:51f98c]  INFO -- : Q-task_id([service_template_provision_task_1000000000412]) Aborting instantiation because [Method exited with rc=MIQ_ABORT]

Comment 2 mkanoor 2017-01-16 17:08:12 UTC
I was able to recreate this outside of our product and it seems to be happening because  we call the exit inside of a loop while traversing remote objects.
DRb is trying to do some kind of cleanup and it ends of throwing the error.

We have filed a bug with the Ruby 
https://bugs.ruby-lang.org/issues/13127

I would like to emphasis that this is not related to setting the retry but rather because we are trying to exit the loop.

#--- Sample Automate Method ---

def remote_vm_array
  vms = $evm.vmdb('vm').all
  vms.each_with_index do |vm, index|
    $evm.log(:info, "Item #{index} : #{vm.name}")
    if index > 5
     exit MIQ_OK
    end
  end
end

begin
  remote_vm_array
ensure
  $evm.log(:info, "Remote Array ensure call")
end

#----------End sample ----------

Comment 3 Josh Carter 2018-09-20 11:55:03 UTC
Bug Closure

Dear customer, 

The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. 

If you have any concerns about this, please let us know.

Thanks and regards!


Note You need to log in before you can comment on or make changes to this bug.