Description of problem: Service Catalog( Orchestration type) is not working in Global Region Version-Release number of selected component (if applicable): 4.6 How reproducible: Always Steps to Reproduce: 1. Create a service catalog with type orchestration template (heat or Aws cloudformation) in remote region and this catalog is provisioned successfully from remote region. 2. If same catalog is ordered from global region, provisioning fails, in logs it shows no values from dialog to automate are passed: [----] I, [2018-12-04T01:16:02.367655 #12676:38a5dd8] INFO -- : MIQ(MiqAeEngine.deliver) Delivering {"dialog_tenant_name"=>nil, "dialog_stack_name"=>nil, "dialog_stack_onfailure"=>nil, "dialog_stack_timeout"=>nil} for object [ServiceTemplate.10000000000001] with state [] to Automate Note: Catalog type openstack works fine in the same setup, problem is with orchestration type catalog. Actual results: From Global region, the catalog is not getting provisioned Expected results: From global region, the ordering of catalog should be successful. Additional info:
What version is being used, currently specified as 5.9.8 which does not exist.
Hi, we have this same Bug. CFME version 5.9.5.3. Besides catalog items of type "orchestration" with heat orchestration or AWS cloudformation, we have the same behaviour also with orchestration item of type "Ansible Tower".
Re-assigning to Appliance team to review central admin feature.
Brandon, please take look. Prasad has a reproducer. Thanks.
Hi Lucy, The customer logs and reproducer environments differ. The customer logs illustrate the issue which is that orchestration stack provisioning should not be executing in the global region. The zone is not important when queuing Service provisions because Services are zoneless. Provider specific provisions executed as part of a service provision are supposed to be queued to the provider zone. Orchestration provisioning is not currently queueing the work, and ends up trying to execute in the global region, which causes the failure. Let's discuss after the break. Customer global region log: [----] I, [2018-11-30T11:32:06.144255 #2046:c89114] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [99000000033846], MiqWorker id: [99000000000029], Zone: [], Role: [ems_operations], Server: [], Ident: [generic], Target id: [], Instance id: [99000000000011], Task id: [], Command: [ServiceTemplateProvisionRequest.create_request_tasks], Timeout: [3600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [4.933982103] seconds [----] I, [2018-11-30T11:32:06.144465 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue#deliver) Message id: [99000000033846], Delivering... [----] I, [2018-11-30T11:32:06.148063 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#create_request_tasks) Creating request task instances for: <Provisioning Service [AWS_LAMP_instance_Global_01] from [AWS_LAMP_instance_Global_01]>... [----] I, [2018-11-30T11:32:06.148217 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#call_automate_event) Raising event [request_starting] to Automate synchronously [----] I, [2018-11-30T11:32:06.161051 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"request_starting", "EventStream::event_stream"=>99000000000132, :event_stream_id=>99000000000132} for object [ServiceTemplateProvisionRequest.99000000000011] with state [] to Automate [----] I, [2018-11-30T11:32:08.964421 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#call_automate_event) Raised event [request_starting] to Automate [----] I, [2018-11-30T11:32:09.058212 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#create_child_tasks) - creating service tasks for service <ServiceTemplateProvisionTask:99000000000015> with parent service <none> [----] I, [2018-11-30T11:32:09.652701 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateOrchestration.automate_result_include_service_template?) Include Service Template <AWS_LAMP_instance_Global_01> : <true> [----] I, [2018-11-30T11:32:09.655022 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) Setting Service Owning User to Name=Administrator, ID=99000000000001, Group to Name=EvmGroup-super_administrator, ID=99000000000002 [----] I, [2018-11-30T11:32:09.719658 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#create_child_tasks) - created <0> service tasks for service <ServiceTemplateProvisionTask:99000000000015> with parent service <none> [----] I, [2018-11-30T11:32:09.736322 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#deliver_to_automate) Queuing Service_Template_Provisioning: [Provisioning [AWS_LAMP_instance_Global_01] for Service [AWS_LAMP_instance_Global_01]]... [----] I, [2018-11-30T11:32:09.766703 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue.put) Message id: [99000000033848], id: [], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>99000000000015, :namespace=>"Cloud/Orchestration/Provisioning/StateMachines", :class_name=>"Provision", :instance_name=>"CatalogItemInitialization", :automate_message=>"create", :attrs=>{"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013}, :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001}] [----] I, [2018-11-30T11:32:09.826084 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue#delivered) Message id: [99000000033846], State: [ok], Delivered in [3.68161333] seconds [----] I, [2018-11-30T11:32:11.930127 #1826:c89114] INFO -- : MIQ(MiqServer#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic, wcount=4, priority=200 [----] I, [2018-11-30T11:32:11.934999 #1826:c89114] INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:server_dequeue=>0.0037949085235595703, :worker_monitor=>1.132441759109497, :worker_dequeue=>0.007845640182495117, :total_time=>1.1444718837738037} [----] I, [2018-11-30T11:32:12.333068 #2064:c89114] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [99000000033847], MiqWorker id: [99000000000031], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionRequest", :object_id=>99000000000011, :attrs=>{:event_type=>"request_approved", "EventStream::event_stream"=>99000000000131, :event_stream_id=>99000000000131}, :instance_name=>"Event", :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001, :automate_message=>nil}], Dequeued in: [6.218998732] seconds [----] I, [2018-11-30T11:32:12.333339 #2064:c89114] INFO -- : MIQ(MiqQueue#deliver) Message id: [99000000033847], Delivering... [----] I, [2018-11-30T11:32:12.340036 #2064:c89114] INFO -- : MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"request_approved", "EventStream::event_stream"=>99000000000131, :event_stream_id=>99000000000131} for object [ServiceTemplateProvisionRequest.99000000000011] with state [] to Automate [----] I, [2018-11-30T11:32:12.437448 #26277:c89114] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [99000000033848], MiqWorker id: [99000000000039], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>99000000000015, :namespace=>"Cloud/Orchestration/Provisioning/StateMachines", :class_name=>"Provision", :instance_name=>"CatalogItemInitialization", :automate_message=>"create", :attrs=>{"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013}, :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001}], Dequeued in: [2.684511831] seconds [----] I, [2018-11-30T11:32:12.437665 #26277:c89114] INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(MiqQueue#deliver) Message id: [99000000033848], Delivering... [----] I, [2018-11-30T11:32:12.443729 #26277:c89114] INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(MiqAeEngine.deliver) Delivering {"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013} for object [ServiceTemplateProvisionTask.99000000000015] with state [] to Automate [----] I, [2018-11-30T11:32:12.584426 #26277:c89114] INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(ServiceTemplateProvisionTask#before_ae_starts) Executing Service_Template_Provisioning request: [Provisioning [AWS_LAMP_instance_Global_01] for Service [AWS_LAMP_instance_Global_01]] [----] I, [2018-11-30T11:32:14.568490 #2064:c89114] INFO -- : MIQ(MiqQueue#delivered) Message id: [99000000033847], State: [ok], Delivered in [2.235150049] seconds
in customer global log [----] E, [2018-11-30T11:33:37.598935 #26277:c7874c] ERROR -- : Q-task_id([service_template_provision_task_99000000000015]) MiqAeServiceModelBase.ar_method raised: <MiqException::MiqOrchestrationProvisionError>: <execution expired> in customer logs (local) [----] E, [2018-11-30T12:19:24.479998 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) The following error occurred during instance method <deploy_orchestration_stack> for AR object <#<ServiceOrchestration id: 1000000000012, name: "Simple_VM_service-20181130-121910", description: "the service catalog for a simple VM", guid: "d573f9d7-d1f3-476c-899e-018e4db708ad", type: "ServiceOrchestration", service_template_id: 1000000000001, options: {:dialog=>{:dialog_tenant_name=>"Test_CloudForms", :dialog_stack_name=>"Stack-RHOSP-Global-01", :dialog_stack_onfailure=>"DO_NOTHING", :dialog_stack_timeout=>nil, :dialog_param_image_id=>"RHEL-7.5", :dialog_param_flavor=>"Rhel-75-bigdisk", :dialog_param_security_group=>"1511_testKP", :dialog_param_key_ssh=>"CF_test_key", :dialog_param_network=>"internal_cf_net", :dialog_param_name=>"simple_instance_xx", "dialog_tenant_name"=>"Test_CloudForms", "dialog_stack_name"=>"Stack-RHOSP-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_param_image_id"=>"RHEL-7.5", "dialog_param_flavor"=>"Rhel-75-bigdisk", "dialog_param_security_group"=>"1511_testKP", "dialog_param_key_ssh"=>"CF_test_key", "dialog_param_network"=>"internal_cf_net", "dialog_param_name"=>"simple_instance_xx"}}, display: true, created_at: "2018-11-30 11:18:57", updated_at: "2018-11-30 11:19:11", evm_owner_id: 1000000000001, miq_group_id: 1000000000002, retired: false, retires_on: nil, retirement_warn: nil, retirement_last_warn: nil, retirement_state: nil, retirement_requester: nil, tenant_id: 1000000000001, ancestry: nil, initiator: "user">> [----] E, [2018-11-30T12:19:24.486722 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) MiqAeServiceModelBase.ar_method raised: <NoMethodError>: <undefined method `start_with?' for :dialog_tenant_name:Symbol> [----] E, [2018-11-30T12:19:24.486847 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) /var/www/miq/vmdb/app/models/service_orchestration/option_converter.rb:31:in `block in stack_parameters' [----] E, [2018-11-30T12:19:24.535991 #2037:41f0438] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod provision> Stack Stack-RHOSP-Global-01 creation failed. Reason: undefined method `start_with?' for :dialog_tenant_name:Symbol [----] I, [2018-11-30T12:19:24.581025 #2037:c47124] INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod [/ManageIQ/Cloud/Orchestration/Provisioning/StateMachines/Methods/provision]> Ending [----] I, [2018-11-30T12:19:24.581216 #2037:c47124] INFO -- : Q-task_id([service_template_provision_task_1000000000019]) Method exited with rc=MIQ_OK
https://github.com/ManageIQ/manageiq/pull/18374 has been made based on the initial discussion. But Bill recommended another way of using the internal state machine to address the issue during the PR review. Currently working on the new solution.
https://github.com/ManageIQ/manageiq-content/pull/500
New commits detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/18f6f21ce26d12c7087ec6d793fe8bc126f558e9 commit 18f6f21ce26d12c7087ec6d793fe8bc126f558e9 Author: Lucy Fu <lufu> AuthorDate: Wed Jan 30 16:58:36 2019 -0500 Commit: Lucy Fu <lufu> CommitDate: Wed Jan 30 16:58:36 2019 -0500 Add OrchestrationTemplateRunner. All provider calls should be placed in the queue with the correct provider zone. Add OrchestrationTemplateRunner to handle the provider calls via MiqQueue. https://bugzilla.redhat.com/show_bug.cgi?id=1656351 app/models/manageiq/providers/cloud_manager/orchestration_template_runner.rb | 165 + 1 file changed, 165 insertions(+) https://github.com/ManageIQ/manageiq/commit/08ab7f42608c25c09e5b9db5228cfb1ac192eb28 commit 08ab7f42608c25c09e5b9db5228cfb1ac192eb28 Author: Lucy Fu <lufu> AuthorDate: Wed Jan 30 16:59:22 2019 -0500 Commit: Lucy Fu <lufu> CommitDate: Wed Jan 30 16:59:22 2019 -0500 Calls OrchestrationTemplateRunner for provider calls. https://bugzilla.redhat.com/show_bug.cgi?id=1656351 app/models/service_orchestration.rb | 75 +- spec/models/service_orchestration_spec.rb | 99 +- 2 files changed, 119 insertions(+), 55 deletions(-)