Bug 1288207

Summary: Heat is unable to delete stack's resources if a 'port' name is given to a resource and not a UUID.
Product: Red Hat OpenStack Reporter: Eric Rich <erich>
Component: openstack-heatAssignee: Rabi Mishra <ramishra>
Status: CLOSED ERRATA QA Contact: Amit Ugol <augol>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: erich, mburns, ramishra, rhel-osp-director-maint, sbaker, shardy, ssainkar, therve, yeylon, zbitter
Target Milestone: z4Keywords: ZStream
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-heat-2015.1.2-8.el7ost Doc Type: Bug Fix
Doc Text:
Previously, the code to infer implicit dependencies between Neutron resources could fail given invalid input, and threw the following error: BadRequest: Invalid input for port_id. Reason: 'test-stack-instance_port-kz2mdpj4wbpq-port-5etr6o77ho7f' is not a valid UUID. This failure could occur after validation and thus recurred every time the stack was loaded. The user was unable to delete stack after an error of this type, since the stack could not be loaded. This patch ignores dependencies for the incorrect resource, so that the resource can be deleted. As a result, user can now delete the stack after the error.
Story Points: ---
Clone Of:
: 1309814 1310878 (view as bug list) Environment:
Last Closed: 2016-02-18 16:41:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1309814, 1310878    
Attachments:
Description Flags
Heat Templates used in testing none

Description Eric Rich 2015-12-03 20:11:35 UTC
Description of problem:

The error presents its self as: (curing a stack creation - that ends with a failure) 

BadRequest: Invalid input for port_id. Reason: 'test-stack-instance_port-kz2mdpj4wbpq-port-5etr6o77ho7f' is not a valid UUID.

...

  instance_port:
    type: ../foundations/network_port.yaml
    depends_on: [security_group, tennant_network]
    properties:
      network_name: {get_attr: [tennant_network, name]}
      subnet_id: {get_attr: [tennant_network, subnet_id]}
      security_group_name: {get_attr: [security_group, name]}

...
  instance_floating_ip:
    type: OS::Neutron::FloatingIP
    depends_on: [instance_port]
    properties:
      floating_network: { get_param: external_network }
      port_id: { get_attr: [instance_port, port_name]}  ## Here is where 
                                                         # the failure is

...

The template results in the error above however, upon stack delete the delete never processes. 

Version-Release number of selected component (if applicable): 7.0
How reproducible: 100% 

Additional info:

The only way, currently found to recover is to delete the resources, and then remove the entries in the database by hand, for the stack.

Comment 2 Eric Rich 2015-12-03 20:40:11 UTC
# heat resource-list test-stack
ERROR: test_instance: networks networks: 0 0: port Value must be a string

^^ It is intersting that no heat resources can be pulled because of the state that this stack is in.

Comment 3 Eric Rich 2015-12-03 21:21:03 UTC
To recover "as a work around" you can delete all the resources "manually" and run the following in mysql: 

> update stack set status = "COMPLETE", status_reason = "Stack DELETE completed successfully", action = "DELETE", deleted_at = now()  where id = "STACK_ID";

- Note: STACK_ID needs to be replaced with your stack ID.

Comment 4 Eric Rich 2015-12-03 21:38:57 UTC
It seems that if you change: 

      port_id: { get_attr: [instance_port, port_name]}

to: 

      port_id: { get_attr: [instance_port, port_resource]}

You end up with a different issue/error but same "hung state" with the stack: 

The error is: 

StackValidationFailed: Property error : test_instance: networks Property error : networks: 0 Property error : 0: port Value must be a string

Resource CREATE failed: StackValidationFailed: Property error : test_instance: networks Property error : networks: 0 Property error : 0: port Value must be a string

https://bugzilla.redhat.com/show_bug.cgi?id=1288207#c2 still applies

Comment 5 Eric Rich 2015-12-03 21:40:27 UTC
Created attachment 1101988 [details]
Heat Templates used in testing

These heat temperates are triggered using the following: 

# heat stack-create -f heat-templates/samples/instance_and_network.yaml test-stack -P "image_name=RHEL7.2"

Comment 6 Eric Rich 2015-12-03 22:14:52 UTC
(In reply to Eric Rich from comment #4)
> It seems that if you change: 
> 
>       port_id: { get_attr: [instance_port, port_name]}
> 
> to: 
> 
>       port_id: { get_attr: [instance_port, port_resource]}
> 
> You end up with a different issue/error but same "hung state" with the
> stack: 
> 
> The error is: 
> 
> StackValidationFailed: Property error : test_instance: networks Property
> error : networks: 0 Property error : 0: port Value must be a string
> 
> Resource CREATE failed: StackValidationFailed: Property error :
> test_instance: networks Property error : networks: 0 Property error : 0:
> port Value must be a string
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1288207#c2 still applies

http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server-prop-networks-*-port does not seem to be correct in stating "Value must be of type neutron.port", based on the error it seems to be expecting a string!

Comment 7 Zane Bitter 2015-12-04 02:22:17 UTC
First of all, there's a typo in network_port.yaml:

  port_resource:
    description: Port Resource
    value: {get_resouce: port}

because it says get_resouce instead of get_resource it's not recognised as an intrinsic function and just returns a map. That's why the error says that the value must be a string.

So both error messages on create are correct.

As for why it's hanging, it's difficult to say without more information, although it is likely fixed already in a later version. I really need:

* The version of heat-engine you're running
* The heat-engine log from /var/log/heat/
* Possibly the journal ("journalctl -u openstack-heat-engine.service")

Comment 8 Eric Rich 2015-12-14 22:55:11 UTC
(In reply to Zane Bitter from comment #7)
> First of all, there's a typo in network_port.yaml:
> 
>   port_resource:
>     description: Port Resource
>     value: {get_resouce: port}
> 
> because it says get_resouce instead of get_resource it's not recognised as
> an intrinsic function and just returns a map. That's why the error says that
> the value must be a string.
> 
> So both error messages on create are correct.

This corrects the issue creating the stack, however the bigger issue that remains is if "such a typo" is so simple to have and miss, it should not render a stack useless, or hung. 

> 
> As for why it's hanging, it's difficult to say without more information,
> although it is likely fixed already in a later version. I really need:
> 
> * The version of heat-engine you're running
> * The heat-engine log from /var/log/heat/
> * Possibly the journal ("journalctl -u openstack-heat-engine.service")

I will attach these logs so that the "hung" issue can be reviewed/corrected. 

To be honest, it simply seems like, more error checking is needed to validate that the stack could be deployed, before deploying it. Is this what 'preview' does?

Comment 12 Rabi Mishra 2016-01-14 07:40:18 UTC
I had a look at the problem.

It seems the typo mentioned above in network_port.yaml is again the issue, when we're trying to show/delete the task.

Server resource uses it for the 'networks' property.

 test_instance:
    depends_on: [ssh_key, tennant_network, security_group, instance_port]
    type: OS::Nova::Server
    properties:
      name: {get_param: instance_name}
      key_name: { get_attr: [ssh_key, key_name] }
      image: {get_param: image_name}
      flavor: {get_param: image_flavor}
      networks:
        - port: {get_attr: [instance_port, port_resource]} >>> here
      availability_zone: {get_param: az}

As 'port_resource' is not resolved and returns a map, 'get_attr' fails, when finding dependencies[1] for the server during delete/show.

[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L953

Comment 17 errata-xmlrpc 2016-02-18 16:41:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-0266.html