This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1260436 - Unable to deploy heat stack from bundle catalog item
Unable to deploy heat stack from bundle catalog item
Status: CLOSED ERRATA
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Automate (Show other bugs)
5.4.0
Unspecified Unspecified
urgent Severity high
: GA
: 5.5.0
Assigned To: Tina Fitzgerald
Pete Savage
: ZStream
Depends On: 1277821
Blocks: 1261087
  Show dependency treegraph
 
Reported: 2015-09-06 16:47 EDT by Jared Deubel
Modified: 2017-09-20 15:22 EDT (History)
15 users (show)

See Also:
Fixed In Version: 5.5.0.8
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1261087 (view as bug list)
Environment:
Last Closed: 2015-12-08 08:29:35 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Automate domain changes for Orchestration provisioning with a bundle (14.40 KB, application/zip)
2015-09-08 14:35 EDT, Greg McCullough
no flags Details
(Updated) Automate domain changes for Orchestration provisioning with a bundle (14.45 KB, application/zip)
2015-09-09 12:40 EDT, Greg McCullough
no flags Details

  None (edit)
Description Jared Deubel 2015-09-06 16:47:53 EDT
Description of problem:

We are able to deploy heat stack from CF as a single catalog items. However we need to deploy VMs to multiple site (different Openstack) and we are trying to deploy them as bundle catalog items which is failing.

I think the due a bad/invalid json format for the Active record Object the server is not accepting it and thereby the update of AR object ServiceTemplateProvisionTask fails.
Need to check as to how they are populating the object.

The following error occurred during instance method <update_and_notify_parent> for AR object <#<ServiceTemplateProvisionTask id: 1000000000684, description: "Provisioning [Multisite-OS-Test] for Service [Multi...", state: "finished", request_type: "clone_to_service", userid: "admin", options: {:dialog=>{"dialog_stack_name"=>"MultisiteOSTest", "dialog_stack_onfailure"=>"ROLLBACK", "dialog_stack_timeout"=>"", "dialog_option_param_name"=>"MultisiteOSTestVM", "request"=>"clone_to_service"}, :workflow_settings=>{:resource_action_id=>1000000000577, :dialog_id=>1000000000023}, :src_id=>1000000000022, :delivered_on=>2015-09-06 05:24:58 UTC, :pass=>0, :parsed_dialog_options=>"---\n0:\n  :stack_name: MultisiteOSTest\n  :dialog_stack_name: MultisiteOSTest\n  :stack_onfailure: ROLLBACK\n  :dialog_stack_onfailure: ROLLBACK\n  :option_param_name: MultisiteOSTestVM\n  :dialog_option_param_name: MultisiteOSTestVM\n", :parsed_dialog_tags=>"--- {}\n"}, created_on: "2015-09-06 05:24:58", updated_on: "2015-09-06 05:26:23", message: "Service Provisioned Successfully", status: "Warn", type: "ServiceTemplateProvisionTask", miq_request_id: 1000000000265, source_id: 1000000000022, source_type: "ServiceTemplate", destination_id: 1000000000310, destination_type: "Service", miq_request_task_id: nil, phase: nil, phase_context: {}>>

============
[----] W, [2015-09-06T15:25:12.376647 #5483:5c3e9c]  WARN -- : Q-task_id([service_template_provision_task_1000000000684]) MIQ(MiqQueue.put)        /var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_engine.rb:49 called with large payload (args: 575 bytes, data: 0 bytes) Message id: [1000000628209],  id: [], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [service_template_provision_task_1000000000684], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [100], State: [ready], Deliver On: [2015-09-06 05:26:12 UTC], Data: [], Args: [{:object_type=>"ServiceTemplateProvisionTask", :object_id=>1000000000684, :namespace=>"Service/Provisioning/StateMachines", :class_name=>"ServiceProvision_Template", :instance_name=>"CatalogBundleInitialization", :automate_message=>"create", :attrs=>{"dialog_stack_name"=>"MultisiteOSTest", "dialog_stack_onfailure"=>"ROLLBACK", "dialog_stack_timeout"=>"", "dialog_option_param_name"=>"MultisiteOSTestVM", "request"=>"clone_to_service"}, :user_id=>1000000000001, :state=>"checkprovisioned", :ae_fsm_started=>nil, :ae_state_started=>"2015-09-06 05:25:11 UTC", :ae_state_retries=>1}]
....
[----] E, [2015-09-06T15:26:24.182385 #5483:9d35a00] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) MIQ(abstract_adapter) Name: [], Message: [PGError: ERROR:  value too long for type character varying(255)
...: UPDATE "miq_requests" SET "request_state" = 'finished', "status" = 'Warn', "message" = 'Expected(201) <=> Actual(400 Bad Request)
excon.error.response
  :body          => "{\"explanation\": \"The server could not comply with the request since it is either malformed or otherwise incorrect.\", \"code\": 400, \"error\": {\"message\": \"The server could not comply with the request since it is either malformed or otherwise incorrect.\", \"traceback\": null, \"type\": \"HTTPBadRequest\"}, \"title\": \"Bad Request\"}"
  :headers       => {
    "Content-Length"         => "321"
    "Content-Type"           => "application/json; charset=UTF-8"
    "Date"                   => "Sun, 06 Sep 2015 05:25:30 GMT"
    "X-Openstack-Request-Id" => "req-de6da573-51b1-45a8-a735-55e69a87b18c"
  }
  :local_address => "10.0.14.10"
  :local_port    => 46871
  :reason_phrase => "Bad Request"
  :remote_ip     => "10.0.14.20"
  :status        => 400
  :status_line   => "HTTP/1.1 400 Bad Request\r\n"
 (MiqException...]
[----] E, [2015-09-06T15:26:24.189066 #5483:9d35a00] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> The following error occurred during instance method <update_and_notify_parent> for AR object <#<ServiceTemplateProvisionTask id: 1000000000684, description: "Provisioning [Multisite-OS-Test] for Service [Multi...", state: "finished", request_type: "clone_to_service", userid: "admin", options: {:dialog=>{"dialog_stack_name"=>"MultisiteOSTest", "dialog_stack_onfailure"=>"ROLLBACK", "dialog_stack_timeout"=>"", "dialog_option_param_name"=>"MultisiteOSTestVM", "request"=>"clone_to_service"}, :workflow_settings=>{:resource_action_id=>1000000000577, :dialog_id=>1000000000023}, :src_id=>1000000000022, :delivered_on=>2015-09-06 05:24:58 UTC, :pass=>0, :parsed_dialog_options=>"---\n0:\n  :stack_name: MultisiteOSTest\n  :dialog_stack_name: MultisiteOSTest\n  :stack_onfailure: ROLLBACK\n  :dialog_stack_onfailure: ROLLBACK\n  :option_param_name: MultisiteOSTestVM\n  :dialog_option_param_name: MultisiteOSTestVM\n", :parsed_dialog_tags=>"--- {}\n"}, created_on: "2015-09-06 05:24:58", updated_on: "2015-09-06 05:26:23", message: "Service Provisioned Successfully", status: "Warn", type: "ServiceTemplateProvisionTask", miq_request_id: 1000000000265, source_id: 1000000000022, source_type: "ServiceTemplate", destination_id: 1000000000310, destination_type: "Service", miq_request_task_id: nil, phase: nil, phase_context: {}>>
[----] E, [2015-09-06T15:26:24.189959 #5483:9d35a00] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> MiqAeServiceModelBase.ar_method raised: <ActiveRecord::StatementInvalid>: <Database statement error encountered>
[----] E, [2015-09-06T15:26:24.190133 #5483:9d35a00] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> /opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:1327:in `async_exec'
.....
[----] E, [2015-09-06T15:26:24.193173 #5483:96d7168] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> <AEMethod task_finished> The following error occurred during method evaluation:
[----] E, [2015-09-06T15:26:24.193762 #5483:96d7168] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> <AEMethod task_finished>   DRb::DRbUnknownError: ActiveRecord::
[----] E, [2015-09-06T15:26:24.194689 #5483:96d7168] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> <AEMethod task_finished>   /opt/rh/ruby200/root/usr/share/ruby/drb/drb.rb:1117:in `method_missing'
[----] E, [2015-09-06T15:26:24.208948 #5483:5c3e9c] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> Method STDERR: /opt/rh/ruby200/root/usr/share/ruby/drb/drb.rb:1117:in `method_missing': ActiveRecord:: (DRb::DRbUnknownError)
[----] E, [2015-09-06T15:26:24.209161 #5483:5c3e9c] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> Method STDERR: from <code: task.finished($evm.inputs['message'])>:6:in `<main>'
[----] I, [2015-09-06T15:26:24.212412 #5483:5c3e9c]  INFO -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> <AEMethod [/ManageIQ/System/CommonMethods/StateMachineMethods/task_finished]> Ending
[----] E, [2015-09-06T15:26:24.213754 #5483:5c3e9c] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> Aborting instantiation (unknown method return code) because [Method exited with rc=Unknown RC: [1]]
[----] E, [2015-09-06T15:26:24.214248 #5483:5c3e9c] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> State=<Finished> running  raised exception: <Method exited with rc=Unknown RC: [1]>
[----] W, [2015-09-06T15:26:24.214416 #5483:5c3e9c]  WARN -- : Q-task_id([service_template_provision_task_1000000000684]) <AutomationEngine> Error in State=[Finished]
[----] E, [2015-09-06T15:26:24.215168 #5483:5c3e9c] ERROR -- : Q-task_id([service_template_provision_task_1000000000684]) MIQ(MiqAeEngine.deliver) Error delivering {"dialog_stack_name"=>"MultisiteOSTest", "dialog_stack_onfailure"=>"ROLLBACK", "dialog_stack_timeout"=>"", "dialog_option_param_name"=>"MultisiteOSTestVM", "request"=>"clone_to_service"} for object [ServiceTemplateProvisionTask.1000000000684] with state [checkprovisioned] to Automate: 
[----] I, [2015-09-06T15:26:24.215268 #5483:5c3e9c]  INFO -- : Q-task_id([service_template_provision_task_1000000000684]) MIQ(ServiceTemplateProvisionTask.after_ae_delivery) ae_result="error"
================================================================================================================








Version-Release number of selected component (if applicable):
5.4.1.0

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 14 Bill Wei 2015-09-07 12:25:08 EDT
The error about the message too long (255) haas been resolved in this issue https://bugzilla.redhat.com/show_bug.cgi?id=1252849.
Comment 20 Greg McCullough 2015-09-08 14:35:53 EDT
Created attachment 1071452 [details]
Automate domain changes for Orchestration provisioning with a bundle

We have determined the issue reported here is because the CatalogBundleInitialization/CatalogItemInitialization methods used with the service state machines were originally designed to work with VM provisioning, before Orchestration provisioning was introduced into the product.  These helper methods were designed to assist in passing parameters from a bundle service down to individual VM provision tasks.  Orchestration provisioning uses a different configuration of tasks to perform provisioning which is not supported by the current methods.

Attached is an updated domain containing a small change to the CatalogItemInitialization method that will allow for the dialog fields to be passed to the Orchestration provision item when used with a bundle.  Additionally, it includes the fix from Bug 1252849 which should be applied to the environment as well to protect against 

Finally, the domain includes a CatalogItemInitialization state-machine instance for the /Cloud/Orchestration/Provisioning which needs to be used for the Orchestration Catalog Item when it is being provisioned as part of a bundled service.  This instance runs the CatalogItemInitialization before running the standard orchestration state-machine steps.

We have tested these changes locally and would like the customer to validate them in their environment.
Comment 22 Greg McCullough 2015-09-09 09:37:09 EDT
Do we have access to the full appliance logs?  The automation log above only shows the bundle service and we also want to see the catalog item.

The fog error with "unknown protocol" is normal.
Comment 23 Felix Dewaleyne 2015-09-09 10:16:05 EDT
the archive is available on the dropbox ftp as 01503978-Current_region_1_default_1000000000001_EVM_1000000000001_20150909_031005_20150909_140938.zip
Comment 24 Greg McCullough 2015-09-09 10:19:07 EDT
In addition to Comment 22 I wanted to add that after importing the automate domain you need to perform the following steps:

1. Ensure the domain where you import the changes is enabled.  (If you import into a new domain it will be disabled by default.)

2. The import includes a CatalogItemInitialization state-machine instance for the /Cloud/Orchestration/Provisioning which needs to be used for the Orchestration Catalog Item when it is being provisioned as part of a bundled service.  This instance runs the CatalogItemInitialization before running the standard orchestration state-machine steps.  This change is what does the work of passing the parameters down to the catalog items.
Comment 26 Greg McCullough 2015-09-09 12:40:55 EDT
Created attachment 1071848 [details]
(Updated) Automate domain changes for Orchestration provisioning with a bundle

Looking into this further we realized that the service dialog was modified to use the "option_0_" prefix on the fields which is used in conjunction with the catalog initialization automate methods to control how field values are pass down to sub-services.

For example: Fields that start with "option_0_" are passed to all services.  Fields starting with "option_1_" would only be passed to catalog items in a bundle with a provision index of 1.  In both cases the field name is stripped of the "option_#_" before passing it down.

If a field does not begin with a prefix like this it will be passed to all services with an unmodified field name.

This is the different between what we were testing and what the customer was using.  The customer had "option_0_stack_name" for the field and we were testing with just "stack_name".

The export has been updated to deal with this difference.  Please have the customer test with the latest export provided with this comment.

Also note, the dialog field could be reverted back to use the original field names without the "option_0_" prefix which would work without these latest changes.
Comment 30 CFME Bot 2015-09-25 10:22:25 EDT
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/1c94f4b608530e0870e46be8f7a7e3484614ce2a

commit 1c94f4b608530e0870e46be8f7a7e3484614ce2a
Author:     Tina Fitzgerald <tfitzger@redhat.com>
AuthorDate: Thu Sep 17 15:17:31 2015 -0400
Commit:     Tina Fitzgerald <tfitzger@redhat.com>
CommitDate: Thu Sep 17 15:17:31 2015 -0400

    Changed CatalogItemInitialization automate method to pass dialog
    fields to the Orchestration provision item when used with a bundle.
    
    CatalogBundleInitialization/CatalogItemInitialization methods were
    designed to work with VM provisioning, before Orchestration provisioning
     was introduced. These methods were designed to assist in passing
    parameters from a bundle service down to individual VM provision tasks.
    Orchestration provisioning uses a different configuration of tasks to
    perform provisioning which is not supported by the current methods.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1260436

 .../Methods.class/__methods__/catalogiteminitialization.rb  | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Comment 31 CFME Bot 2015-09-29 12:21:31 EDT
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=68ff7034a90e798a1deac12a221a095c84a79084

commit 68ff7034a90e798a1deac12a221a095c84a79084
Author:     Tina Fitzgerald <tfitzger@redhat.com>
AuthorDate: Thu Sep 17 15:17:31 2015 -0400
Commit:     Tina Fitzgerald <tfitzger@redhat.com>
CommitDate: Mon Sep 28 09:56:30 2015 -0400

    Changed CatalogItemInitialization automate method to pass dialog
    fields to the Orchestration provision item when used with a bundle.
    
    CatalogBundleInitialization/CatalogItemInitialization methods were
    designed to work with VM provisioning, before Orchestration provisioning
     was introduced. These methods were designed to assist in passing
    parameters from a bundle service down to individual VM provision tasks.
    Orchestration provisioning uses a different configuration of tasks to
    perform provisioning which is not supported by the current methods.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1260436

 .../Methods.class/__methods__/catalogiteminitialization.rb  | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Comment 32 CFME Bot 2015-09-29 12:21:44 EDT
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=cba77ec2888c0f59b594795ffc862bc3b5419f7d

commit cba77ec2888c0f59b594795ffc862bc3b5419f7d
Merge: b421bf0 8e836ec
Author:     Greg McCullough <gmccullo@redhat.com>
AuthorDate: Tue Sep 29 12:03:48 2015 -0400
Commit:     Greg McCullough <gmccullo@redhat.com>
CommitDate: Tue Sep 29 12:03:48 2015 -0400

    Merge branch 'bz_1261087' into '5.4.z'
    
    Automate | Services | Changed CatalogItemInitialization automate method for Orchestration provisioning
    
    Applied Orchestration automate fixes.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1261087
    
    Original:
    https://github.com/ManageIQ/manageiq/pull/4416
    https://bugzilla.redhat.com/show_bug.cgi?id=1260436
    
    Clean cherry-pick except for spec test. Applying spec test patch wasn't successful since test was created after vmdb was rerooted.
    
    
    See merge request !266

 .../Provision.class/catalogiteminitization.yaml    |  14 +++
 .../__methods__/catalogiteminitialization.rb       |  15 +++
 .../catalog_item_initialization_spec.rb            | 105 +++++++++++++++++++++
 vmdb/spec/support/service_template_helper.rb       |  96 +++++++++++++++++++
 4 files changed, 230 insertions(+)
Comment 33 CFME Bot 2015-09-29 12:48:30 EDT
New commit detected on cfme_productization/master:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme_productization.git;a=commit;h=68ff7034a90e798a1deac12a221a095c84a79084

commit 68ff7034a90e798a1deac12a221a095c84a79084
Author:     Tina Fitzgerald <tfitzger@redhat.com>
AuthorDate: Thu Sep 17 15:17:31 2015 -0400
Commit:     Tina Fitzgerald <tfitzger@redhat.com>
CommitDate: Mon Sep 28 09:56:30 2015 -0400

    Changed CatalogItemInitialization automate method to pass dialog
    fields to the Orchestration provision item when used with a bundle.
    
    CatalogBundleInitialization/CatalogItemInitialization methods were
    designed to work with VM provisioning, before Orchestration provisioning
     was introduced. These methods were designed to assist in passing
    parameters from a bundle service down to individual VM provision tasks.
    Orchestration provisioning uses a different configuration of tasks to
    perform provisioning which is not supported by the current methods.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1260436

 .../Methods.class/__methods__/catalogiteminitialization.rb  | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Comment 34 CFME Bot 2015-09-29 12:48:42 EDT
New commit detected on cfme_productization/master:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme_productization.git;a=commit;h=cba77ec2888c0f59b594795ffc862bc3b5419f7d

commit cba77ec2888c0f59b594795ffc862bc3b5419f7d
Merge: b421bf0 8e836ec
Author:     Greg McCullough <gmccullo@redhat.com>
AuthorDate: Tue Sep 29 12:03:48 2015 -0400
Commit:     Greg McCullough <gmccullo@redhat.com>
CommitDate: Tue Sep 29 12:03:48 2015 -0400

    Merge branch 'bz_1261087' into '5.4.z'
    
    Automate | Services | Changed CatalogItemInitialization automate method for Orchestration provisioning
    
    Applied Orchestration automate fixes.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1261087
    
    Original:
    https://github.com/ManageIQ/manageiq/pull/4416
    https://bugzilla.redhat.com/show_bug.cgi?id=1260436
    
    Clean cherry-pick except for spec test. Applying spec test patch wasn't successful since test was created after vmdb was rerooted.
    
    
    See merge request !266

 .../Provision.class/catalogiteminitization.yaml    |  14 +++
 .../__methods__/catalogiteminitialization.rb       |  15 +++
 .../catalog_item_initialization_spec.rb            | 105 +++++++++++++++++++++
 vmdb/spec/support/service_template_helper.rb       |  96 +++++++++++++++++++
 4 files changed, 230 insertions(+)
Comment 35 Pete Savage 2015-11-20 07:59:03 EST
To verify....
Added two RHOS providers, one RHOS6 one RHOS7
Added a stack template
Created a catalog item for each provider and pointed them to the orchestration entrypoint
Created a bundle and pointed it to the service entrypoint
Deployed service

Verified working in 5.5.0.11
Comment 37 errata-xmlrpc 2015-12-08 08:29:35 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551

Note You need to log in before you can comment on or make changes to this bug.