Bug 1656351
Summary: | Service Catalog( Orchestration type) is not working in Global Region | |||
---|---|---|---|---|
Product: | Red Hat CloudForms Management Engine | Reporter: | Neha Chugh <nchugh> | |
Component: | Appliance | Assignee: | Lucy Fu <lufu> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Niyaz Akhtar Ansari <nansari> | |
Severity: | high | Docs Contact: | Red Hat CloudForms Documentation <cloudforms-docs> | |
Priority: | high | |||
Version: | 5.9.4 | CC: | abellott, akarol, bmidwood, dmetzger, giovanni.dellacorte, gmccullo, lavenel, lufu, mkanoor, mnadeem, nansari, nchugh, obarenbo, pmukhedk, simaishi, tfitzger, wfitzger | |
Target Milestone: | GA | Keywords: | TestOnly, ZStream | |
Target Release: | 5.11.0 | |||
Hardware: | All | |||
OS: | All | |||
Whiteboard: | ||||
Fixed In Version: | 5.11.0.1 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1686891 (view as bug list) | Environment: | ||
Last Closed: | 2019-12-13 15:01:44 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | Bug | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | CFME Core | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1728179 | |||
Bug Blocks: | 1686891 |
Description
Neha Chugh
2018-12-05 10:11:55 UTC
What version is being used, currently specified as 5.9.8 which does not exist. Hi, we have this same Bug. CFME version 5.9.5.3. Besides catalog items of type "orchestration" with heat orchestration or AWS cloudformation, we have the same behaviour also with orchestration item of type "Ansible Tower". Re-assigning to Appliance team to review central admin feature. Re-assigning to Appliance team to review central admin feature. Brandon, please take look. Prasad has a reproducer. Thanks. Hi Lucy, The customer logs and reproducer environments differ. The customer logs illustrate the issue which is that orchestration stack provisioning should not be executing in the global region. The zone is not important when queuing Service provisions because Services are zoneless. Provider specific provisions executed as part of a service provision are supposed to be queued to the provider zone. Orchestration provisioning is not currently queueing the work, and ends up trying to execute in the global region, which causes the failure. Let's discuss after the break. Customer global region log: [----] I, [2018-11-30T11:32:06.144255 #2046:c89114] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [99000000033846], MiqWorker id: [99000000000029], Zone: [], Role: [ems_operations], Server: [], Ident: [generic], Target id: [], Instance id: [99000000000011], Task id: [], Command: [ServiceTemplateProvisionRequest.create_request_tasks], Timeout: [3600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [4.933982103] seconds [----] I, [2018-11-30T11:32:06.144465 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue#deliver) Message id: [99000000033846], Delivering... [----] I, [2018-11-30T11:32:06.148063 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#create_request_tasks) Creating request task instances for: <Provisioning Service [AWS_LAMP_instance_Global_01] from [AWS_LAMP_instance_Global_01]>... [----] I, [2018-11-30T11:32:06.148217 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#call_automate_event) Raising event [request_starting] to Automate synchronously [----] I, [2018-11-30T11:32:06.161051 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"request_starting", "EventStream::event_stream"=>99000000000132, :event_stream_id=>99000000000132} for object [ServiceTemplateProvisionRequest.99000000000011] with state [] to Automate [----] I, [2018-11-30T11:32:08.964421 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionRequest#call_automate_event) Raised event [request_starting] to Automate [----] I, [2018-11-30T11:32:09.058212 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#create_child_tasks) - creating service tasks for service <ServiceTemplateProvisionTask:99000000000015> with parent service <none> [----] I, [2018-11-30T11:32:09.652701 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateOrchestration.automate_result_include_service_template?) Include Service Template <AWS_LAMP_instance_Global_01> : <true> [----] I, [2018-11-30T11:32:09.655022 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) Setting Service Owning User to Name=Administrator, ID=99000000000001, Group to Name=EvmGroup-super_administrator, ID=99000000000002 [----] I, [2018-11-30T11:32:09.719658 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#create_child_tasks) - created <0> service tasks for service <ServiceTemplateProvisionTask:99000000000015> with parent service <none> [----] I, [2018-11-30T11:32:09.736322 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(ServiceTemplateProvisionTask#deliver_to_automate) Queuing Service_Template_Provisioning: [Provisioning [AWS_LAMP_instance_Global_01] for Service [AWS_LAMP_instance_Global_01]]... [----] I, [2018-11-30T11:32:09.766703 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue.put) Message id: [99000000033848], id: [], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>99000000000015, :namespace=>"Cloud/Orchestration/Provisioning/StateMachines", :class_name=>"Provision", :instance_name=>"CatalogItemInitialization", :automate_message=>"create", :attrs=>{"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013}, :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001}] [----] I, [2018-11-30T11:32:09.826084 #2046:c89114] INFO -- : Q-task_id([service_template_provision_request_99000000000011]) MIQ(MiqQueue#delivered) Message id: [99000000033846], State: [ok], Delivered in [3.68161333] seconds [----] I, [2018-11-30T11:32:11.930127 #1826:c89114] INFO -- : MIQ(MiqServer#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic, wcount=4, priority=200 [----] I, [2018-11-30T11:32:11.934999 #1826:c89114] INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:server_dequeue=>0.0037949085235595703, :worker_monitor=>1.132441759109497, :worker_dequeue=>0.007845640182495117, :total_time=>1.1444718837738037} [----] I, [2018-11-30T11:32:12.333068 #2064:c89114] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [99000000033847], MiqWorker id: [99000000000031], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionRequest", :object_id=>99000000000011, :attrs=>{:event_type=>"request_approved", "EventStream::event_stream"=>99000000000131, :event_stream_id=>99000000000131}, :instance_name=>"Event", :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001, :automate_message=>nil}], Dequeued in: [6.218998732] seconds [----] I, [2018-11-30T11:32:12.333339 #2064:c89114] INFO -- : MIQ(MiqQueue#deliver) Message id: [99000000033847], Delivering... [----] I, [2018-11-30T11:32:12.340036 #2064:c89114] INFO -- : MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"request_approved", "EventStream::event_stream"=>99000000000131, :event_stream_id=>99000000000131} for object [ServiceTemplateProvisionRequest.99000000000011] with state [] to Automate [----] I, [2018-11-30T11:32:12.437448 #26277:c89114] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [99000000033848], MiqWorker id: [99000000000039], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>99000000000015, :namespace=>"Cloud/Orchestration/Provisioning/StateMachines", :class_name=>"Provision", :instance_name=>"CatalogItemInitialization", :automate_message=>"create", :attrs=>{"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013}, :user_id=>99000000000001, :miq_group_id=>99000000000002, :tenant_id=>99000000000001}], Dequeued in: [2.684511831] seconds [----] I, [2018-11-30T11:32:12.437665 #26277:c89114] INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(MiqQueue#deliver) Message id: [99000000033848], Delivering... [----] I, [2018-11-30T11:32:12.443729 #26277:c89114] INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(MiqAeEngine.deliver) Delivering {"dialog_tenant_name"=>nil, "dialog_stack_name"=>"Stack-AWS-CF-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_stack_notifications"=>nil, "dialog_stack_capabilities"=>"CAPABILITY_IAM", "dialog_stack_resource_types"=>nil, "dialog_stack_role"=>nil, "dialog_stack_tags"=>nil, "dialog_stack_policy"=>nil, "dialog_param_KeyName"=>"shift-aws", "dialog_param_DBName"=>"MyDatabase", "password::dialog_param_DBUser"=>"********", "password::dialog_param_DBPassword"=>"********", "password::dialog_param_DBRootPassword"=>"********", "dialog_param_InstanceType"=>"t2.small", "dialog_param_SSHLocation"=>"0.0.0.0/0", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000013} for object [ServiceTemplateProvisionTask.99000000000015] with state [] to Automate [----] I, [2018-11-30T11:32:12.584426 #26277:c89114] INFO -- : Q-task_id([service_template_provision_task_99000000000015]) MIQ(ServiceTemplateProvisionTask#before_ae_starts) Executing Service_Template_Provisioning request: [Provisioning [AWS_LAMP_instance_Global_01] for Service [AWS_LAMP_instance_Global_01]] [----] I, [2018-11-30T11:32:14.568490 #2064:c89114] INFO -- : MIQ(MiqQueue#delivered) Message id: [99000000033847], State: [ok], Delivered in [2.235150049] seconds in customer global log [----] E, [2018-11-30T11:33:37.598935 #26277:c7874c] ERROR -- : Q-task_id([service_template_provision_task_99000000000015]) MiqAeServiceModelBase.ar_method raised: <MiqException::MiqOrchestrationProvisionError>: <execution expired> in customer logs (local) [----] E, [2018-11-30T12:19:24.479998 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) The following error occurred during instance method <deploy_orchestration_stack> for AR object <#<ServiceOrchestration id: 1000000000012, name: "Simple_VM_service-20181130-121910", description: "the service catalog for a simple VM", guid: "d573f9d7-d1f3-476c-899e-018e4db708ad", type: "ServiceOrchestration", service_template_id: 1000000000001, options: {:dialog=>{:dialog_tenant_name=>"Test_CloudForms", :dialog_stack_name=>"Stack-RHOSP-Global-01", :dialog_stack_onfailure=>"DO_NOTHING", :dialog_stack_timeout=>nil, :dialog_param_image_id=>"RHEL-7.5", :dialog_param_flavor=>"Rhel-75-bigdisk", :dialog_param_security_group=>"1511_testKP", :dialog_param_key_ssh=>"CF_test_key", :dialog_param_network=>"internal_cf_net", :dialog_param_name=>"simple_instance_xx", "dialog_tenant_name"=>"Test_CloudForms", "dialog_stack_name"=>"Stack-RHOSP-Global-01", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_param_image_id"=>"RHEL-7.5", "dialog_param_flavor"=>"Rhel-75-bigdisk", "dialog_param_security_group"=>"1511_testKP", "dialog_param_key_ssh"=>"CF_test_key", "dialog_param_network"=>"internal_cf_net", "dialog_param_name"=>"simple_instance_xx"}}, display: true, created_at: "2018-11-30 11:18:57", updated_at: "2018-11-30 11:19:11", evm_owner_id: 1000000000001, miq_group_id: 1000000000002, retired: false, retires_on: nil, retirement_warn: nil, retirement_last_warn: nil, retirement_state: nil, retirement_requester: nil, tenant_id: 1000000000001, ancestry: nil, initiator: "user">> [----] E, [2018-11-30T12:19:24.486722 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) MiqAeServiceModelBase.ar_method raised: <NoMethodError>: <undefined method `start_with?' for :dialog_tenant_name:Symbol> [----] E, [2018-11-30T12:19:24.486847 #2037:c418a0] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) /var/www/miq/vmdb/app/models/service_orchestration/option_converter.rb:31:in `block in stack_parameters' [----] E, [2018-11-30T12:19:24.535991 #2037:41f0438] ERROR -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod provision> Stack Stack-RHOSP-Global-01 creation failed. Reason: undefined method `start_with?' for :dialog_tenant_name:Symbol [----] I, [2018-11-30T12:19:24.581025 #2037:c47124] INFO -- : Q-task_id([service_template_provision_task_1000000000019]) <AEMethod [/ManageIQ/Cloud/Orchestration/Provisioning/StateMachines/Methods/provision]> Ending [----] I, [2018-11-30T12:19:24.581216 #2037:c47124] INFO -- : Q-task_id([service_template_provision_task_1000000000019]) Method exited with rc=MIQ_OK https://github.com/ManageIQ/manageiq/pull/18374 has been made based on the initial discussion. But Bill recommended another way of using the internal state machine to address the issue during the PR review. Currently working on the new solution. New commits detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/18f6f21ce26d12c7087ec6d793fe8bc126f558e9 commit 18f6f21ce26d12c7087ec6d793fe8bc126f558e9 Author: Lucy Fu <lufu> AuthorDate: Wed Jan 30 16:58:36 2019 -0500 Commit: Lucy Fu <lufu> CommitDate: Wed Jan 30 16:58:36 2019 -0500 Add OrchestrationTemplateRunner. All provider calls should be placed in the queue with the correct provider zone. Add OrchestrationTemplateRunner to handle the provider calls via MiqQueue. https://bugzilla.redhat.com/show_bug.cgi?id=1656351 app/models/manageiq/providers/cloud_manager/orchestration_template_runner.rb | 165 + 1 file changed, 165 insertions(+) https://github.com/ManageIQ/manageiq/commit/08ab7f42608c25c09e5b9db5228cfb1ac192eb28 commit 08ab7f42608c25c09e5b9db5228cfb1ac192eb28 Author: Lucy Fu <lufu> AuthorDate: Wed Jan 30 16:59:22 2019 -0500 Commit: Lucy Fu <lufu> CommitDate: Wed Jan 30 16:59:22 2019 -0500 Calls OrchestrationTemplateRunner for provider calls. https://bugzilla.redhat.com/show_bug.cgi?id=1656351 app/models/service_orchestration.rb | 75 +- spec/models/service_orchestration_spec.rb | 99 +- 2 files changed, 119 insertions(+), 55 deletions(-) |