Bug 1656351 - Service Catalog( Orchestration type) is not working in Global Region
Summary: Service Catalog( Orchestration type) is not working in Global Region
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.9.4
Hardware: All
OS: All
high
high
Target Milestone: GA
: 5.11.0
Assignee: Lucy Fu
QA Contact: Niyaz Akhtar Ansari
Red Hat CloudForms Documentation
URL:
Whiteboard:
Depends On: 1728179
Blocks: 1686891
TreeView+ depends on / blocked
 
Reported: 2018-12-05 10:11 UTC by Neha Chugh
Modified: 2019-12-13 15:01 UTC (History)
17 users (show)

Fixed In Version: 5.11.0.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1686891 (view as bug list)
Environment:
Last Closed: 2019-12-13 15:01:44 UTC
Category: Bug
Cloudforms Team: CFME Core
Target Upstream Version:


Attachments (Terms of Use)

Description Neha Chugh 2018-12-05 10:11:55 UTC
Description of problem:
Service Catalog( Orchestration type) is not working in Global Region

Version-Release number of selected component (if applicable):
4.6

How reproducible:
Always

Steps to Reproduce:

1. Create a service catalog with type orchestration template (heat or Aws cloudformation) in remote region and this catalog is provisioned successfully from remote region.
2. If same catalog is ordered from global region, provisioning fails, in logs it shows no values from dialog to automate are passed: 

[----] I, [2018-12-04T01:16:02.367655 #12676:38a5dd8]  INFO -- : MIQ(MiqAeEngine.deliver) Delivering {"dialog_tenant_name"=>nil, "dialog_stack_name"=>nil, "dialog_stack_onfailure"=>nil, "dialog_stack_timeout"=>nil} for object [ServiceTemplate.10000000000001] with state [] to Automate

Note: Catalog type openstack works fine in the same setup, problem is with orchestration type catalog.

Actual results:
From Global region, the catalog is not getting provisioned

Expected results:
From global region, the ordering of catalog should be successful.

Additional info:

Comment 2 dmetzger 2018-12-05 13:15:59 UTC
What version is being used, currently specified as 5.9.8 which does not exist.

Comment 5 giovanni.dellacorte 2018-12-11 08:23:34 UTC
Hi, 

we have this same Bug.
CFME version 5.9.5.3.

Besides catalog items of type "orchestration" with heat orchestration or AWS cloudformation, we have the same behaviour also with orchestration item of type "Ansible Tower".

Comment 6 Greg McCullough 2018-12-13 21:25:46 UTC
Re-assigning to Appliance team to review central admin feature.

Comment 7 Greg McCullough 2018-12-13 21:26:05 UTC
Re-assigning to Appliance team to review central admin feature.

Comment 9 Gregg Tanzillo 2018-12-18 11:20:27 UTC
Brandon, please take look. Prasad has a reproducer. Thanks.

Comment 11 Tina Fitzgerald 2018-12-21 18:41:39 UTC
Hi Lucy,

The customer logs and reproducer environments differ. The customer logs illustrate the issue which is that orchestration stack provisioning should not be executing in the global region.

The zone is not important when queuing Service provisions because Services are zoneless. Provider specific provisions executed as part of a 
 service provision are supposed to be queued to the provider zone.  Orchestration provisioning is not currently queueing the work, and ends up trying to execute in the global region, which causes the failure.    

Let's discuss after the break.


Customer global region log:

[----] I, [2018-11-30T11:32:06.144255 #2046:c89114]  INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [99000000033846], MiqWorker id: [99000000000029], Zone: [], Role: [ems_operations], Server: [], Ident: [generic], Target id: [], Instance id: [99000000000011], Task id: [], Command: [ServiceTemplateProvisionRequest.create_request_tasks], Timeout: [3600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [4.933982103] seconds
[----] I, [2018-11-30T11:32:06.144465 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue#deliver) Message id: [99000000033846], Delivering...
[----] I, [2018-11-30T11:32:06.148063 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#create_request_tasks) Creating request task instances for: <Provisioning Service [AWS_LAMP_instance_Global_01] from [AWS_LAMP_instance_Global_01]>...
[----] I, [2018-11-30T11:32:06.148217 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#call_automate_event) Raising event [request_starting] to Automate synchronously
[----] I, [2018-11-30T11:32:06.161051 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"request_starting", "EventStream::event_stream"=>99000000000132, :event_stream_id=>99000000000132} for object [ServiceTemplateProvisionRequest.99000000000011] with state [] to Automate
[----] I, [2018-11-30T11:32:08.964421 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#call_automate_event) Raised event [request_starting] to Automate
[----] I, [2018-11-30T11:32:09.058212 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#create_child_tasks) - creating service tasks for service <ServiceTemplateProvisionTask:99000000000015> with parent service <none>
[----] I, [2018-11-30T11:32:09.652701 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateOrchestration.automate_result_include_service_template?) Include Service Template <AWS_LAMP_instance_Global_01> : <true>
[----] I, [2018-11-30T11:32:09.655022 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) Setting Service Owning User to Name=Administrator, ID=99000000000001, Group to Name=EvmGroup-super_administrator, ID=99000000000002
[----] I, [2018-11-30T11:32:09.719658 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#create_child_tasks) - created <0> service tasks for service <ServiceTemplateProvisionTask:99000000000015> with parent service <none>
[----] I, [2018-11-30T11:32:09.736322 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#deliver_to_automate) Queuing Service_Template_Provisioning: [Provisioning [AWS_LAMP_instance_Global_01] for Service [AWS_LAMP_instance_Global_01]]...
[----] I, [2018-11-30T11:32:09.766703 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue.put) Message id: [99000000033848],  id: [], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>99000000000015, :namespace=>"Cloud/Orchestration/Provisioning/StateMachines", :class_name=>"Provision", :instance_name=>"CatalogItemInitialization", :automate_message=>"create", :attrs=>{"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013}, :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001}]
[----] I, [2018-11-30T11:32:09.826084 #2046:c89114]  INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue#delivered) Message id: [99000000033846], State: [ok], Delivered in [3.68161333] seconds
[----] I, [2018-11-30T11:32:11.930127 #1826:c89114]  INFO -- : MIQ(MiqServer#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic, wcount=4, priority=200
[----] I, [2018-11-30T11:32:11.934999 #1826:c89114]  INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:server_dequeue=>0.0037949085235595703, :worker_monitor=>1.132441759109497, :worker_dequeue=>0.007845640182495117, :total_time=>1.1444718837738037}
[----] I, [2018-11-30T11:32:12.333068 #2064:c89114]  INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [99000000033847], MiqWorker id: [99000000000031], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionRequest", :object_id=>99000000000011, :attrs=>{:event_type=>"request_approved", "EventStream::event_stream"=>99000000000131, :event_stream_id=>99000000000131}, :instance_name=>"Event", :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001, :automate_message=>nil}], Dequeued in: [6.218998732] seconds
[----] I, [2018-11-30T11:32:12.333339 #2064:c89114]  INFO -- : MIQ(MiqQueue#deliver) Message id: [99000000033847], Delivering...
[----] I, [2018-11-30T11:32:12.340036 #2064:c89114]  INFO -- : MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"request_approved", "EventStream::event_stream"=>99000000000131, :event_stream_id=>99000000000131} for object [ServiceTemplateProvisionRequest.99000000000011] with state [] to Automate
[----] I, [2018-11-30T11:32:12.437448 #26277:c89114]  INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [99000000033848], MiqWorker id: [99000000000039], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>99000000000015, :namespace=>"Cloud/Orchestration/Provisioning/StateMachines", :class_name=>"Provision", :instance_name=>"CatalogItemInitialization", :automate_message=>"create", :attrs=>{"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013}, :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001}], Dequeued in: [2.684511831] seconds
[----] I, [2018-11-30T11:32:12.437665 #26277:c89114]  INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(MiqQueue#deliver) Message id: [99000000033848], Delivering...
[----] I, [2018-11-30T11:32:12.443729 #26277:c89114]  INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(MiqAeEngine.deliver) Delivering {"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013} for object [ServiceTemplateProvisionTask.99000000000015] with state [] to Automate
[----] I, [2018-11-30T11:32:12.584426 #26277:c89114]  INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(ServiceTemplateProvisionTask#before_ae_starts) Executing Service_Template_Provisioning request: [Provisioning [AWS_LAMP_instance_Global_01] for Service [AWS_LAMP_instance_Global_01]]
[----] I, [2018-11-30T11:32:14.568490 #2064:c89114]  INFO -- : MIQ(MiqQueue#delivered) Message id: [99000000033847], State: [ok], Delivered in [2.235150049] seconds

Comment 13 William Fitzgerald 2019-01-03 20:02:48 UTC
in customer global log

[----] E, [2018-11-30T11:33:37.598935 #26277:c7874c] ERROR -- : Q-task_id([service_template_provision_task_99000000000015]) MiqAeServiceModelBase.ar_method raised: <MiqException::MiqOrchestrationProvisionError>: <execution expired>


in customer logs (local)


[----] E, [2018-11-30T12:19:24.479998 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) The following error occurred during instance method <deploy_orchestration_stack> for AR object <#<ServiceOrchestration id: 1000000000012, name: "Simple_VM_service-20181130-121910", description: "the service catalog for a simple VM", guid: "d573f9d7-d1f3-476c-899e-018e4db708ad", type: "ServiceOrchestration", service_template_id: 1000000000001, options: {:dialog=>{:dialog_tenant_name=>"Test_CloudForms", :dialog_stack_name=>"Stack-RHOSP-Global-01", :dialog_stack_onfailure=>"DO_NOTHING", :dialog_stack_timeout=>nil, :dialog_param_image_id=>"RHEL-7.5", :dialog_param_flavor=>"Rhel-75-bigdisk", :dialog_param_security_group=>"1511_testKP", :dialog_param_key_ssh=>"CF_test_key", :dialog_param_network=>"internal_cf_net", :dialog_param_name=>"simple_instance_xx", "dialog_tenant_name"=>"Test_CloudForms", "dialog_stack_name"=>"Stack-RHOSP-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_param_image_id"=>"RHEL-7.5", "dialog_param_flavor"=>"Rhel-75-bigdisk", "dialog_param_security_group"=>"1511_testKP", "dialog_param_key_ssh"=>"CF_test_key", "dialog_param_network"=>"internal_cf_net", "dialog_param_name"=>"simple_instance_xx"}}, display: true, created_at: "2018-11-30 11:18:57", updated_at: "2018-11-30 11:19:11", evm_owner_id: 1000000000001, miq_group_id: 1000000000002, retired: false, retires_on: nil, retirement_warn: nil, retirement_last_warn: nil, retirement_state: nil, retirement_requester: nil, tenant_id: 1000000000001, ancestry: nil, initiator: "user">>
[----] E, [2018-11-30T12:19:24.486722 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) MiqAeServiceModelBase.ar_method raised: <NoMethodError>: <undefined method `start_with?' for :dialog_tenant_name:Symbol>
[----] E, [2018-11-30T12:19:24.486847 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) /var/www/miq/vmdb/app/models/service_orchestration/option_converter.rb:31:in `block in stack_parameters'
[----] E, [2018-11-30T12:19:24.535991 #2037:41f0438] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod provision> Stack Stack-RHOSP-Global-01 creation failed. Reason: undefined method `start_with?' for :dialog_tenant_name:Symbol
[----] I, [2018-11-30T12:19:24.581025 #2037:c47124]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod [/ManageIQ/Cloud/Orchestration/Provisioning/StateMachines/Methods/provision]> Ending
[----] I, [2018-11-30T12:19:24.581216 #2037:c47124]  INFO -- : Q-task_id([service_template_provision_task_1000000000019]) Method exited with rc=MIQ_OK

Comment 15 Lucy Fu 2019-01-24 16:44:39 UTC
https://github.com/ManageIQ/manageiq/pull/18374 has been made based on the initial discussion. But Bill recommended another way of using the internal state machine to address the issue during the PR review. Currently working on the new solution.

Comment 17 CFME Bot 2019-03-05 23:15:57 UTC
New commits detected on ManageIQ/manageiq/master:

https://github.com/ManageIQ/manageiq/commit/18f6f21ce26d12c7087ec6d793fe8bc126f558e9
commit 18f6f21ce26d12c7087ec6d793fe8bc126f558e9
Author:     Lucy Fu <lufu@redhat.com>
AuthorDate: Wed Jan 30 16:58:36 2019 -0500
Commit:     Lucy Fu <lufu@redhat.com>
CommitDate: Wed Jan 30 16:58:36 2019 -0500

    Add OrchestrationTemplateRunner.

    All provider calls should be placed in the queue with the correct provider zone.
    Add OrchestrationTemplateRunner to handle the provider calls via MiqQueue.

    https://bugzilla.redhat.com/show_bug.cgi?id=1656351

 app/models/manageiq/providers/cloud_manager/orchestration_template_runner.rb | 165 +
 1 file changed, 165 insertions(+)


https://github.com/ManageIQ/manageiq/commit/08ab7f42608c25c09e5b9db5228cfb1ac192eb28
commit 08ab7f42608c25c09e5b9db5228cfb1ac192eb28
Author:     Lucy Fu <lufu@redhat.com>
AuthorDate: Wed Jan 30 16:59:22 2019 -0500
Commit:     Lucy Fu <lufu@redhat.com>
CommitDate: Wed Jan 30 16:59:22 2019 -0500

    Calls OrchestrationTemplateRunner for provider calls.

    https://bugzilla.redhat.com/show_bug.cgi?id=1656351

 app/models/service_orchestration.rb | 75 +-
 spec/models/service_orchestration_spec.rb | 99 +-
 2 files changed, 119 insertions(+), 55 deletions(-)


Note You need to log in before you can comment on or make changes to this bug.