Bug 1573339

Summary: Unable to suspend stack with alarm resource
Product: Red Hat OpenStack Reporter: Zane Bitter <zbitter>
Component: openstack-aodhAssignee: Zane Bitter <zbitter>
Status: CLOSED EOL QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: apevec, jschluet, lhh, mburns, panburaj, ramishra, rhel-osp-director-maint, sbaker, shardy, slinaber, srevivo, ssmolyak, therve, zbitter
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1558461
: 1573340 (view as bug list) Environment:
Last Closed: 2018-06-22 12:38:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1558461    
Bug Blocks: 1573340    

Description Zane Bitter 2018-04-30 22:02:52 UTC 
+++ This bug was initially created as a clone of Bug #1558461 +++

Description of problem:

When I try to suspend a stack that has alarm in it, it gives me below error.
~~~
2018-03-19 12:03:14Z [Stack02.cpu_alarm_high]: SUSPEND_FAILED  BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
2018-03-19 12:03:14Z [Stack02]: SUSPEND_FAILED  Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
~~~


Version-Release number of selected component (if applicable):

[stack@instack ~]$ rpm -qa | grep heat
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-common-7.0.6-1.el7ost.noarch
openstack-heat-api-7.0.6-1.el7ost.noarch
openstack-tripleo-heat-templates-5.3.3-1.el7ost.noarch
puppet-heat-9.5.0-2.el7ost.noarch
python-heatclient-1.5.2-1.el7ost.noarch
openstack-heat-engine-7.0.6-1.el7ost.noarch
openstack-heat-templates-0-0.14.1e6015dgit.el7ost.noarch
python-heat-agent-0-0.14.1e6015dgit.el7ost.noarch
openstack-heat-api-cfn-7.0.6-1.el7ost.noarch


How reproducible:
100%


Steps to Reproduce:

1. Create a auto-scaling stack with alarms.
Snippet of template:
~~~
..
..
    cpu_alarm_high:
        type: OS::Aodh::GnocchiAggregationByResourcesAlarm
        properties:
            description: Scale-up if the average CPU > x% for y minute
            metric: cpu_util
            aggregation_method: mean
            granularity: { get_param: time_range }
            evaluation_periods: { get_param: cpu_up_period_time }
            threshold: { get_param: max_cpu }
            resource_type: instance
            comparison_operator: gt
            alarm_actions:
                - str_replace:
                    template: trust+url
                    params:
                        url: {get_attr: [server_scaleup_policy, signal_url]}
            query:
              list_join:
                - ''
                - - {'=': {server_group: {get_param: "OS::stack_id"}}}
..
..
~~~


2. Suspend the stack using the below command.
~~~
openstack stack suspend Stack02
~~~


3. Stack suspend fails with below error.

[stack@instack ~]$ openstack stack list
+--------------------------------------+------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+-----------------+----------------------+--------------+
| eb15d973-09ce-4af8-8c74-9b86e8ba75df | Stack02    | SUSPEND_FAILED  | 2018-03-19T11:30:37Z | None         |
| db0fff87-0a0a-41fa-9466-6a948dcefabd | Stack01    | RESUME_COMPLETE | 2018-03-19T07:44:44Z | None         |
+--------------------------------------+------------+-----------------+----------------------+--------------+


[stack@instack ~]$ openstack stack show Stack02
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                                                     |
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                    | eb15d973-09ce-4af8-8c74-9b86e8ba75df                                                                                                                                      |
| stack_name            | Stack02                                                                                                                                                                   |
| description           | Simple template to deploy a single compute instance                                                                                                                       |
| creation_time         | 2018-03-19T11:30:37Z                                                                                                                                                      |
| updated_time          | None                                                                                                                                                                      |
| stack_status          | SUSPEND_FAILED                                                                                                                                                            |
| stack_status_reason   | Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)     |
| parameters            | OS::project_id: a10de11a32df479d90df59dbadfb5433                                                                                                                          |
..
..


[stack@instack ~]$ openstack stack event list Stack02
2018-03-19 11:30:38Z [Stack02]: CREATE_IN_PROGRESS  Stack CREATE started
2018-03-19 11:30:38Z [Stack02.lb]: CREATE_IN_PROGRESS  state changed
2018-03-19 11:30:40Z [Stack02.lb]: CREATE_COMPLETE  state changed
..
..
2018-03-19 11:31:29Z [Stack02]: CREATE_COMPLETE  Stack CREATE completed successfully
2018-03-19 12:03:14Z [Stack02]: SUSPEND_IN_PROGRESS  Stack SUSPEND started
2018-03-19 12:03:14Z [Stack02.monitor]: SUSPEND_IN_PROGRESS  state changed
2018-03-19 12:03:14Z [Stack02.monitor]: SUSPEND_COMPLETE  state changed
2018-03-19 12:03:14Z [Stack02.cpu_alarm_high]: SUSPEND_IN_PROGRESS  state changed
2018-03-19 12:03:14Z [Stack02.cpu_alarm_high]: SUSPEND_FAILED  BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
2018-03-19 12:03:14Z [Stack02]: SUSPEND_FAILED  Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)


[root@overcloud-controller-0 ~]# tail /var/log/heat/heat-engine.log -n25
2018-03-19 12:03:14.050 402232 INFO heat.engine.resource [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] suspending HealthMonitor "monitor" [364b819d-36ec-4566-928a-291e1c842ea0] Stack "Stack02" [eb15d973-09ce-4af8-8c74-9b86e8ba75df]
2018-03-19 12:03:14.101 402232 INFO heat.engine.resource [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] suspending AodhGnocchiAggregationByResourcesAlarm "cpu_alarm_high" [497fce25-9179-4b21-a833-f735a1a093fc] Stack "Stack02" [eb15d973-09ce-4af8-8c74-9b86e8ba75df]
2018-03-19 12:03:14.938 402232 INFO heat.engine.resource [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] SUSPEND: AodhGnocchiAggregationByResourcesAlarm "cpu_alarm_high" [497fce25-9179-4b21-a833-f735a1a093fc] Stack "Stack02" [eb15d973-09ce-4af8-8c74-9b86e8ba75df]
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource Traceback (most recent call last):
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 770, in _action_recorder
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     yield
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 872, in _do_action
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 336, in wrapper
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     step = next(subtask)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 817, in action_handler_task
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     handler_data = handler(*args)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/alarm_base.py", line 202, in handle_suspend
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     alarm_update)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/aodhclient/v2/alarm.py", line 156, in update
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     data=jsonutils.dumps(alarm)).json()
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/aodhclient/v2/base.py", line 45, in _put
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     return self.client.api.put(*args, **kwargs)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 196, in put
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     return self.request(url, 'PUT', **kwargs)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/aodhclient/client.py", line 38, in request
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     raise exceptions.from_response(resp, url, method)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource BadRequest: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource 
2018-03-19 12:03:14.993 402232 INFO heat.engine.stack [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] Stack SUSPEND FAILED (Stack02): Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)


Actual results:
Suspend fails for stack with alarm all the time.


Expected results:
Suspend should be successful.

--- Additional comment from Zane Bitter on 2018-03-22 10:06:56 EDT ---

It looks a lot like this upstream bug: https://bugs.launchpad.net/aodh/+bug/1737186

--- Additional comment from Zane Bitter on 2018-04-24 16:20:53 EDT ---

Fix is merged in upstream master (Rocky) and stable/queens.
Comment 1 Scott Lewis 2018-06-22 12:38:15 UTC 
OSP11 is now retired, see details at https://access.redhat.com/errata/product/191/ver=11/rhel---7/x86_64/RHBA-2018:1828