Description of problem: While using a global region to provision a service that relies on data determined in a dialog, a provisioning error is raised Version-Release number of selected component (if applicable): 5.9.5.3 How reproducible: all the time Steps to Reproduce: 1.create two cloudforms regions, central and remote 2.set up sync between them 3.set up a provider and dialog to provision a service on remote 4.test service provision on remote 5.test service provision on central Actual results: on remote the evm.log shows [----] E, [2018-11-23T09:42:28.354907 #11837:5e4d7c] ERROR -- : Q-task_id([service_template_provision_task_1000000000001]) MiqAeServiceModelBase.ar_method raised: <NoMethodError>: <undefined method `start_with?' for :dialog_tenant_name:Symbol> [----] E, [2018-11-23T09:42:28.355029 #11837:5e4d7c] ERROR -- : Q-task_id([service_template_provision_task_1000000000001]) /var/www/miq/vmdb/app/models/service_orchestration/option_converter.rb:31:in `block in stack_parameters' /var/www/miq/vmdb/app/models/service_orchestration/option_converter.rb:30:in `each' /var/www/miq/vmdb/app/models/service_orchestration/option_converter.rb:30:in `stack_parameters' /opt/rh/cfme-gemset/bundler/gems/cfme-providers-openstack-73905e8a8376/app/models/manageiq/providers/openstack/cloud_manager/orchestration_service_option_converter.rb:6:in `stack_create_options' /var/www/miq/vmdb/app/models/service_orchestration.rb:57:in `build_stack_options_from_dialog' /var/www/miq/vmdb/app/models/service_orchestration.rb:119:in `build_stack_create_options' /var/www/miq/vmdb/app/models/mixins/service_orchestration_options_mixin.rb:4:in `stack_options' /var/www/miq/vmdb/app/models/service_orchestration.rb:148:in `save_create_options' /var/www/miq/vmdb/app/models/service_orchestration.rb:34:in `ensure in deploy_orchestration_stack' /var/www/miq/vmdb/app/models/service_orchestration.rb:34:in `deploy_orchestration_stack' Expected results: the provision is made successfully Additional info:
Re-assigning to Appliance team to review central admin feature.
Lucy - Please review the automation.log to help identify when the duplicate symbol keys are added to the options hash that Joe pointed out in comment #8.
This log message from global region has dialog options keys in string format which is expected. [----] I, [2018-11-23T10:13:04.644429 #12453:1052e30] INFO -- : Q-task_id([service_template_provision_task_99000000000001]) <AEMethod dialog_parser> dialog_options: {"dialog_tenant_name"=>"Test_CloudForms", "dialog_stack_name"=>"SimpleVMfromGLOBAL-00", "dialog_stack_onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_param_image_id"=>"RHEL-7.5", "dialog_param_flavor"=>"Rhel-75-bigdisk", "dialog_param_security_group"=>"CF_segroup", "dialog_param_key_ssh"=>"1511_testKP", "dialog_param_network"=>"internal_cf_net", "dialog_param_name"=>"simple_instance_glbl-01", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000001} This message from remote region has dialog options keys in symbol format which is wrong. [----] I, [2018-11-23T09:42:14.494859 #11837:4d98d6c] INFO -- : Q-task_id([service_template_provision_task_1000000000001]) <AEMethod dialog_parser> dialog_options: {:dialog_tenant_name=>"Test_CloudForms", :dialog_stack_name=>"SImple-VM-launch-Global-test-00", :dialog_stack_onfailure=>"DO_NOTHING", :dialog_stack_timeout=>nil, :dialog_param_image_id=>"RHEL-7.5", :dialog_param_flavor=>"Rhel-75-bigdisk", :dialog_param_security_group=>"CF_segroup", :dialog_param_key_ssh=>"1511_testKP", :dialog_param_network=>"internal_cf_net", :dialog_param_name=>"simple_instance_glbl_01", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>1000000000001} This dialog options data in the log message is retrieved from VMDB which was collected by service dialog during service ordering and saved into VMDB. Don't understand why the data from remote region is in symbol format and from global region in string format. Is the customer using different version in global and remote appliances? Is there any customization to the code base in remote region?
Hi, since we raised this case, I can answer to the questions by Lucy. Is the customer using different version in global and remote appliances? NO, we are using CFME 5.9.5 for both global and remote CloudForms regions Is there any customization to the code base in remote region? NO, it is the provided appliance, without any modification. Regards, Giovanni
Please run the following commands from rails console and send us the data. ServiceTemplateProvisionTask.find(1000000000001) ServiceTemplateProvisionTask.find(99000000000001) Thanks.
irb(main):004:0* ServiceTemplateProvisionTask.find(1000000000001) PostgreSQLAdapter#log_after_checkout, connection_pool: size: 5, connections: 1, in use: 1, waiting_in_queue: 0 => #<ServiceTemplateProvisionTask id: 1000000000001, description: "Provisioning [Simple_VM_service] for Service [Simp...", state: "finished", request_type: " clone_to_service", userid: "admin", options: {:dialog=>{:dialog_tenant_name=>"Test_CloudForms", :dialog_stack_name=>"SImple-VM-launch-Global-test-00", :dialo g_stack_onfailure=>"DO_NOTHING", :dialog_stack_timeout=>nil, :dialog_param_image_id=>"RHEL-7.5", :dialog_param_flavor=>"Rhel-75-bigdisk", :dialog_param_secur ity_group=>"CF_segroup", :dialog_param_key_ssh=>"1511_testKP", :dialog_param_network=>"internal_cf_net", :dialog_param_name=>"simple_instance_glbl_01", "requ est"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>1000000000001}, :workflow_settings=>{:resource_action_id=>1000000000031, :dialog_ id=>1000000000002}, :initiator=>nil, :src_id=>1000000000001, :request_options=>{:submit_workflow=>true, :init_defaults=>false}, :cart_state=>"ordered", :requ ester_group=>"EvmGroup-super_administrator", :delivered_on=>2018-11-23 08:42:01 UTC, :pass=>0, :parsed_dialog_options=>"---\n0:\n :tenant_name: Test_CloudFo rms\n :dialog_tenant_name: Test_CloudForms\n :stack_name: SImple-VM-launch-Global-test-00\n :dialog_stack_name: SImple-VM-launch-Global-test-00\n :stack_ onfailure: DO_NOTHING\n :dialog_stack_onfailure: DO_NOTHING\n :param_image_id: RHEL-7.5\n :dialog_param_image_id: RHEL-7.5\n :param_flavor: Rhel-75-bigdi sk\n :dialog_param_flavor: Rhel-75-bigdisk\n :param_security_group: CF_segroup\n :dialog_param_security_group: CF_segroup\n :param_key_ssh: 1511_testKP\n :dialog_param_key_ssh: 1511_testKP\n :param_network: internal_cf_net\n :dialog_param_network: internal_cf_net\n :param_name: simple_instance_glbl_01\n :dialog_param_name: simple_instance_glbl_01\n", :parsed_dialog_tags=>"--- {}\n"}, created_on: "2018-11-23 08:42:01", updated_on: "2018-11-23 08:42:29", messa ge: "Service_Template_Provisioning failed", status: "Error", type: "ServiceTemplateProvisionTask", miq_request_id: 1000000000001, source_id: 1000000000001, s ource_type: "ServiceTemplate", destination_id: 1000000000001, destination_type: "Service", miq_request_task_id: nil, phase: nil, phase_context: {}, tenant_id : 1000000000001> irb(main):005:0> ServiceTemplateProvisionTask.find(99000000000001) => #<ServiceTemplateProvisionTask id: 99000000000001, description: "Provisioning [Simple_Instance_from_Global] for Ser...", state: "finished", request_type: "clone_to_service", userid: "admin", options: {:dialog=>{"dialog_tenant_name"=>"Test_CloudForms", "dialog_stack_name"=>"SimpleVMfromGLOBAL-00", "dialog_stack _onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_param_image_id"=>"RHEL-7.5", "dialog_param_flavor"=>"Rhel-75-bigdisk", "dialog_param_security _group"=>"CF_segroup", "dialog_param_key_ssh"=>"1511_testKP", "dialog_param_network"=>"internal_cf_net", "dialog_param_name"=>"simple_instance_glbl-01", "req uest"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000001}, :workflow_settings=>{:resource_action_id=>99000000000021, :dial og_id=>99000000000002}, :initiator=>nil, :src_id=>99000000000001, :request_options=>{:submit_workflow=>true, :init_defaults=>false}, :cart_state=>"ordered", :requester_group=>"EvmGroup-super_administrator", :delivered_on=>2018-11-23 09:12:49 UTC, :pass=>0, :parsed_dialog_options=>"---\n0:\n :tenant_name: Test_Cl oudForms\n :dialog_tenant_name: Test_CloudForms\n :stack_name: SimpleVMfromGLOBAL-00\n :dialog_stack_name: SimpleVMfromGLOBAL-00\n :stack_onfailure: DO_N OTHING\n :dialog_stack_onfailure: DO_NOTHING\n :param_image_id: RHEL-7.5\n :dialog_param_image_id: RHEL-7.5\n :param_flavor: Rhel-75-bigdisk\n :dialog_p aram_flavor: Rhel-75-bigdisk\n :param_security_group: CF_segroup\n :dialog_param_security_group: CF_segroup\n :param_key_ssh: 1511_testKP\n :dialog_param _key_ssh: 1511_testKP\n :param_network: internal_cf_net\n :dialog_param_network: internal_cf_net\n :param_name: simple_instance_glbl-01\n :dialog_param_n ame: simple_instance_glbl-01\n", :parsed_dialog_tags=>"--- {}\n"}, created_on: "2018-11-23 09:12:49", updated_on: "2018-11-23 10:17:43", message: "Service_Te mplate_Provisioning failed", status: "Error", type: "ServiceTemplateProvisionTask", miq_request_id: 99000000000001, source_id: 99000000000001, source_type: " ServiceTemplate", destination_id: 99000000000001, destination_type: "Service", miq_request_task_id: nil, phase: nil, phase_context: {}, tenant_id: 9900000000 0001> Both taken at the Global Region CF Appliance.
irb(main):004:0* ServiceTemplateProvisionTask.find(1000000000001) PostgreSQLAdapter#log_after_checkout, connection_pool: size: 5, connections: 1, in use: 1, waiting_in_queue: 0 => #<ServiceTemplateProvisionTask id: 1000000000001, description: "Provisioning [Simple_VM_service] for Service [Simp...", state: "finished", request_type: " clone_to_service", userid: "admin", options: {:dialog=>{:dialog_tenant_name=>"Test_CloudForms", :dialog_stack_name=>"SImple-VM-launch-Global-test-00", :dialo ^in symbol ^in symbol ^ g_stack_onfailure=>"DO_NOTHING", :dialog_stack_timeout=>nil, :dialog_param_image_id=>"RHEL-7.5", :dialog_param_flavor=>"Rhel-75-bigdisk", :dialog_param_secur ity_group=>"CF_segroup", :dialog_param_key_ssh=>"1511_testKP", :dialog_param_network=>"internal_cf_net", :dialog_param_name=>"simple_instance_glbl_01", "requ est"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>1000000000001}, :workflow_settings=>{:resource_action_id=>1000000000031, :dialog_ id=>1000000000002}, :initiator=>nil, :src_id=>1000000000001, :request_options=>{:submit_workflow=>true, :init_defaults=>false}, :cart_state=>"ordered", :requ ester_group=>"EvmGroup-super_administrator", :delivered_on=>2018-11-23 08:42:01 UTC, :pass=>0, :parsed_dialog_options=>"---\n0:\n :tenant_name: Test_CloudFo rms\n :dialog_tenant_name: Test_CloudForms\n :stack_name: SImple-VM-launch-Global-test-00\n :dialog_stack_name: SImple-VM-launch-Global-test-00\n :stack_ onfailure: DO_NOTHING\n :dialog_stack_onfailure: DO_NOTHING\n :param_image_id: RHEL-7.5\n :dialog_param_image_id: RHEL-7.5\n :param_flavor: Rhel-75-bigdi sk\n :dialog_param_flavor: Rhel-75-bigdisk\n :param_security_group: CF_segroup\n :dialog_param_security_group: CF_segroup\n :param_key_ssh: 1511_testKP\n :dialog_param_key_ssh: 1511_testKP\n :param_network: internal_cf_net\n :dialog_param_network: internal_cf_net\n :param_name: simple_instance_glbl_01\n :dialog_param_name: simple_instance_glbl_01\n", :parsed_dialog_tags=>"--- {}\n"}, created_on: "2018-11-23 08:42:01", updated_on: "2018-11-23 08:42:29", messa ge: "Service_Template_Provisioning failed", status: "Error", type: "ServiceTemplateProvisionTask", miq_request_id: 1000000000001, source_id: 1000000000001, s ource_type: "ServiceTemplate", destination_id: 1000000000001, destination_type: "Service", miq_request_task_id: nil, phase: nil, phase_context: {}, tenant_id : 1000000000001> irb(main):005:0> ServiceTemplateProvisionTask.find(99000000000001) => #<ServiceTemplateProvisionTask id: 99000000000001, description: "Provisioning [Simple_Instance_from_Global] for Ser...", state: "finished", request_type: "clone_to_service", userid: "admin", options: {:dialog=>{"dialog_tenant_name"=>"Test_CloudForms", "dialog_stack_name"=>"SimpleVMfromGLOBAL-00", "dialog_stack ^in string ^string ^string _onfailure"=>"DO_NOTHING", "dialog_stack_timeout"=>nil, "dialog_param_image_id"=>"RHEL-7.5", "dialog_param_flavor"=>"Rhel-75-bigdisk", "dialog_param_security _group"=>"CF_segroup", "dialog_param_key_ssh"=>"1511_testKP", "dialog_param_network"=>"internal_cf_net", "dialog_param_name"=>"simple_instance_glbl-01", "req uest"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>99000000000001}, :workflow_settings=>{:resource_action_id=>99000000000021, :dial og_id=>99000000000002}, :initiator=>nil, :src_id=>99000000000001, :request_options=>{:submit_workflow=>true, :init_defaults=>false}, :cart_state=>"ordered", :requester_group=>"EvmGroup-super_administrator", :delivered_on=>2018-11-23 09:12:49 UTC, :pass=>0, :parsed_dialog_options=>"---\n0:\n :tenant_name: Test_Cl oudForms\n :dialog_tenant_name: Test_CloudForms\n :stack_name: SimpleVMfromGLOBAL-00\n :dialog_stack_name: SimpleVMfromGLOBAL-00\n :stack_onfailure: DO_N OTHING\n :dialog_stack_onfailure: DO_NOTHING\n :param_image_id: RHEL-7.5\n :dialog_param_image_id: RHEL-7.5\n :param_flavor: Rhel-75-bigdisk\n :dialog_p aram_flavor: Rhel-75-bigdisk\n :param_security_group: CF_segroup\n :dialog_param_security_group: CF_segroup\n :param_key_ssh: 1511_testKP\n :dialog_param _key_ssh: 1511_testKP\n :param_network: internal_cf_net\n :dialog_param_network: internal_cf_net\n :param_name: simple_instance_glbl-01\n :dialog_param_n ame: simple_instance_glbl-01\n", :parsed_dialog_tags=>"--- {}\n"}, created_on: "2018-11-23 09:12:49", updated_on: "2018-11-23 10:17:43", message: "Service_Te mplate_Provisioning failed", status: "Error", type: "ServiceTemplateProvisionTask", miq_request_id: 99000000000001, source_id: 99000000000001, source_type: " ServiceTemplate", destination_id: 99000000000001, destination_type: "Service", miq_request_task_id: nil, phase: nil, phase_context: {}, tenant_id: 9900000000 0001> When data retrieved from VMDB, the dialog options from remote region is in symbol format which is wrong and it is in string from global region. Wondering why the data format in DB is different if they all run with the same code.
It looks like this change is possibly the problem: https://github.com/ManageIQ/manageiq-api/pull/312 Before that change, the keys of the option hash was only symbolized the first level deep, but another client expected symbols all the way, so it was changed to deep_symbolize_keys. See the original issue that caused the above change: https://github.com/ManageIQ/manageiq-api/issues/260 Finally, there are many places in the api trying to standardize on symbol keys for options hashes. If we were to change how requests work, it would conflict with what we're doing elsewhere. Will need to review our options. app/controllers/api/actions_controller.rb: data["options"] = data["options"].deep_symbolize_keys if data["options"] app/controllers/api/actions_controller.rb: data["options"] = data["options"].deep_symbolize_keys if data["options"] app/controllers/api/alert_definitions_controller.rb: super(type, id, data.deep_symbolize_keys).serializable_hash app/controllers/api/alert_definitions_controller.rb: super(type, id, data.deep_symbolize_keys) app/controllers/api/authentications_controller.rb: task_id = auth.update_in_provider_queue(data.deep_symbolize_keys) app/controllers/api/configuration_script_sources_controller.rb: task_id = config_script_src.update_in_provider_queue(data.deep_symbolize_keys) app/controllers/api/configuration_script_sources_controller.rb: task_id = klass.create_in_provider_queue(manager.id, data.except('manager_resource').deep_symbolize_keys) app/controllers/api/custom_button_sets_controller.rb: super(type, id, data.deep_symbolize_keys) app/controllers/api/custom_button_sets_controller.rb: super(type, id, data.deep_symbolize_keys) app/controllers/api/custom_buttons_controller.rb: custom_button.options = data["options"].deep_symbolize_keys if data["options"] app/controllers/api/custom_buttons_controller.rb: custom_button.visibility = data["visibility"].deep_symbolize_keys if data["visibility"] app/controllers/api/custom_buttons_controller.rb: custom_button.update_attributes!(updated_data.deep_symbolize_keys) if data.present? app/controllers/api/generic_object_definitions_controller.rb: klass.create!(data.deep_symbolize_keys) app/controllers/api/generic_object_definitions_controller.rb: go_def.update_attributes!(updated_data.deep_symbolize_keys) if data.present? app/controllers/api/providers_controller.rb: data["options"] = data["options"].deep_symbolize_keys if data.key?("options") app/controllers/api/service_templates_controller.rb: catalog_item_type.create_catalog_item(data.deep_symbolize_keys, User.current_user.userid) app/controllers/api/service_templates_controller.rb: catalog_item.update_catalog_item(data.deep_symbolize_keys, User.current_user.userid) app/controllers/api/subcollections/cloud_subnets.rb: data.deep_symbolize_keys! app/controllers/api/subcollections/request_tasks.rb: updated_options = request_task.options.merge(Hash(data.fetch("options")).deep_symbolize_keys) app/controllers/api/subcollections/schedules.rb: BaseController.instance_method(:edit_resource).bind(self).call(:schedules, id, data.deep_symbolize_keys) app/controllers/api/users_controller.rb: data["settings"] = Hash(current_settings).deep_merge(settings.deep_symbolize_keys) lib/services/api/authentication_service.rb: klass.create_in_provider_queue(manager_resource.id, attrs.deep_symbolize_keys) lib/services/api/request_parser.rb: data["options"].deep_symbolize_keys spec/requests/alert_definitions_spec.rb: expect(alert_definition.expression).to eq(sample_alert_definition["hash_expression"].deep_symbolize_keys) spec/requests/providers_spec.rb: options_symbolized = options.deep_symbolize_keys
We're currently trying an approach of letting the api create the records as it currently does but allow the readers of these objects get a HashWithIndifferentAccess which will allow either string or symbol access into the hash. diff --git a/app/models/mixins/service_mixin.rb b/app/models/mixins/service_mixin.rb index 03cfe6851d..4435599364 100644 --- a/app/models/mixins/service_mixin.rb +++ b/app/models/mixins/service_mixin.rb @@ -11,6 +11,10 @@ module ServiceMixin include UuidMixin include OwnershipMixin acts_as_miq_taggable + + def options + self.options = super.with_indifferent_access + end end If that works, we can then try to normalize the clients of these objects over time to follow the same access so that there aren't mixed symbol / string hash access.
Lucy created the following PR to address the issue: https://github.com/ManageIQ/manageiq/pull/18373
PR was merged, setting to POST.
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/5cf587ca97ccd752aa1e0261397b2c75d502b48a commit 5cf587ca97ccd752aa1e0261397b2c75d502b48a Author: Lucy Fu <lufu> AuthorDate: Fri Jan 18 11:59:02 2019 -0500 Commit: Lucy Fu <lufu> CommitDate: Fri Jan 18 11:59:02 2019 -0500 Let stack_parameters support symbol hash keys, for now Service dialog options keys are supposed to be in String. But for some reason, API sends those keys in symbol when ordering a Service from global region. Before we find out where those keys are duplicated in symbol via API, we have to work around it. https://bugzilla.redhat.com/show_bug.cgi?id=1654999 app/models/service_orchestration/option_converter.rb | 2 +- spec/models/service_orchestration_spec.rb | 14 +- 2 files changed, 13 insertions(+), 3 deletions(-)
Verified in 5.11.0.11