Bug 1288207 - Heat is unable to delete stack's resources if a 'port' name is given to a resource and not a UUID.
Heat is unable to delete stack's resources if a 'port' name is given to a res...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity high
: z4
: 7.0 (Kilo)
Assigned To: Rabi Mishra
Amit Ugol
: ZStream
Depends On:
Blocks: 1309814 1310878
  Show dependency treegraph
 
Reported: 2015-12-03 15:11 EST by Eric Rich
Modified: 2016-04-26 10:05 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-heat-2015.1.2-8.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the code to infer implicit dependencies between Neutron resources could fail given invalid input, and threw the following error: BadRequest: Invalid input for port_id. Reason: 'test-stack-instance_port-kz2mdpj4wbpq-port-5etr6o77ho7f' is not a valid UUID. This failure could occur after validation and thus recurred every time the stack was loaded. The user was unable to delete stack after an error of this type, since the stack could not be loaded. This patch ignores dependencies for the incorrect resource, so that the resource can be deleted. As a result, user can now delete the stack after the error.
Story Points: ---
Clone Of:
: 1309814 1310878 (view as bug list)
Environment:
Last Closed: 2016-02-18 11:41:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Heat Templates used in testing (2.91 KB, application/x-xz)
2015-12-03 16:40 EST, Eric Rich
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1536515 None None None 2016-01-28 13:34 EST
Red Hat Knowledge Base (Solution) 2089301 None None None Never
OpenStack gerrit 272373 None None None 2016-01-28 13:34 EST
Red Hat Product Errata RHSA-2016:0266 normal SHIPPED_LIVE Moderate: openstack-heat bug fix and security advisory 2016-02-18 16:41:02 EST

  None (edit)
Description Eric Rich 2015-12-03 15:11:35 EST
Description of problem:

The error presents its self as: (curing a stack creation - that ends with a failure) 

BadRequest: Invalid input for port_id. Reason: 'test-stack-instance_port-kz2mdpj4wbpq-port-5etr6o77ho7f' is not a valid UUID.

...

  instance_port:
    type: ../foundations/network_port.yaml
    depends_on: [security_group, tennant_network]
    properties:
      network_name: {get_attr: [tennant_network, name]}
      subnet_id: {get_attr: [tennant_network, subnet_id]}
      security_group_name: {get_attr: [security_group, name]}

...
  instance_floating_ip:
    type: OS::Neutron::FloatingIP
    depends_on: [instance_port]
    properties:
      floating_network: { get_param: external_network }
      port_id: { get_attr: [instance_port, port_name]}  ## Here is where 
                                                         # the failure is

...

The template results in the error above however, upon stack delete the delete never processes. 

Version-Release number of selected component (if applicable): 7.0
How reproducible: 100% 

Additional info:

The only way, currently found to recover is to delete the resources, and then remove the entries in the database by hand, for the stack.
Comment 2 Eric Rich 2015-12-03 15:40:11 EST
# heat resource-list test-stack
ERROR: test_instance: networks networks: 0 0: port Value must be a string

^^ It is intersting that no heat resources can be pulled because of the state that this stack is in.
Comment 3 Eric Rich 2015-12-03 16:21:03 EST
To recover "as a work around" you can delete all the resources "manually" and run the following in mysql: 

> update stack set status = "COMPLETE", status_reason = "Stack DELETE completed successfully", action = "DELETE", deleted_at = now()  where id = "STACK_ID";

- Note: STACK_ID needs to be replaced with your stack ID.
Comment 4 Eric Rich 2015-12-03 16:38:57 EST
It seems that if you change: 

      port_id: { get_attr: [instance_port, port_name]}

to: 

      port_id: { get_attr: [instance_port, port_resource]}

You end up with a different issue/error but same "hung state" with the stack: 

The error is: 

StackValidationFailed: Property error : test_instance: networks Property error : networks: 0 Property error : 0: port Value must be a string

Resource CREATE failed: StackValidationFailed: Property error : test_instance: networks Property error : networks: 0 Property error : 0: port Value must be a string

https://bugzilla.redhat.com/show_bug.cgi?id=1288207#c2 still applies
Comment 5 Eric Rich 2015-12-03 16:40 EST
Created attachment 1101988 [details]
Heat Templates used in testing

These heat temperates are triggered using the following: 

# heat stack-create -f heat-templates/samples/instance_and_network.yaml test-stack -P "image_name=RHEL7.2"
Comment 6 Eric Rich 2015-12-03 17:14:52 EST
(In reply to Eric Rich from comment #4)
> It seems that if you change: 
> 
>       port_id: { get_attr: [instance_port, port_name]}
> 
> to: 
> 
>       port_id: { get_attr: [instance_port, port_resource]}
> 
> You end up with a different issue/error but same "hung state" with the
> stack: 
> 
> The error is: 
> 
> StackValidationFailed: Property error : test_instance: networks Property
> error : networks: 0 Property error : 0: port Value must be a string
> 
> Resource CREATE failed: StackValidationFailed: Property error :
> test_instance: networks Property error : networks: 0 Property error : 0:
> port Value must be a string
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1288207#c2 still applies

http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server-prop-networks-*-port does not seem to be correct in stating "Value must be of type neutron.port", based on the error it seems to be expecting a string!
Comment 7 Zane Bitter 2015-12-03 21:22:17 EST
First of all, there's a typo in network_port.yaml:

  port_resource:
    description: Port Resource
    value: {get_resouce: port}

because it says get_resouce instead of get_resource it's not recognised as an intrinsic function and just returns a map. That's why the error says that the value must be a string.

So both error messages on create are correct.

As for why it's hanging, it's difficult to say without more information, although it is likely fixed already in a later version. I really need:

* The version of heat-engine you're running
* The heat-engine log from /var/log/heat/
* Possibly the journal ("journalctl -u openstack-heat-engine.service")
Comment 8 Eric Rich 2015-12-14 17:55:11 EST
(In reply to Zane Bitter from comment #7)
> First of all, there's a typo in network_port.yaml:
> 
>   port_resource:
>     description: Port Resource
>     value: {get_resouce: port}
> 
> because it says get_resouce instead of get_resource it's not recognised as
> an intrinsic function and just returns a map. That's why the error says that
> the value must be a string.
> 
> So both error messages on create are correct.

This corrects the issue creating the stack, however the bigger issue that remains is if "such a typo" is so simple to have and miss, it should not render a stack useless, or hung. 

> 
> As for why it's hanging, it's difficult to say without more information,
> although it is likely fixed already in a later version. I really need:
> 
> * The version of heat-engine you're running
> * The heat-engine log from /var/log/heat/
> * Possibly the journal ("journalctl -u openstack-heat-engine.service")

I will attach these logs so that the "hung" issue can be reviewed/corrected. 

To be honest, it simply seems like, more error checking is needed to validate that the stack could be deployed, before deploying it. Is this what 'preview' does?
Comment 12 Rabi Mishra 2016-01-14 02:40:18 EST
I had a look at the problem.

It seems the typo mentioned above in network_port.yaml is again the issue, when we're trying to show/delete the task.

Server resource uses it for the 'networks' property.

 test_instance:
    depends_on: [ssh_key, tennant_network, security_group, instance_port]
    type: OS::Nova::Server
    properties:
      name: {get_param: instance_name}
      key_name: { get_attr: [ssh_key, key_name] }
      image: {get_param: image_name}
      flavor: {get_param: image_flavor}
      networks:
        - port: {get_attr: [instance_port, port_resource]} >>> here
      availability_zone: {get_param: az}

As 'port_resource' is not resolved and returns a map, 'get_attr' fails, when finding dependencies[1] for the server during delete/show.

[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L953
Comment 17 errata-xmlrpc 2016-02-18 11:41:55 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-0266.html

Note You need to log in before you can comment on or make changes to this bug.