Bug 1558461

Summary: Unable to suspend stack with alarm resource
Product: Red Hat OpenStack Reporter: Petersingh Anburaj <panburaj>
Component: openstack-aodhAssignee: Zane Bitter <zbitter>
Status: CLOSED CURRENTRELEASE QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: alonare, apevec, augol, jamsmith, jschluet, kiyyappa, lhh, mburns, mmagr, pkilambi, ramishra, rhel-osp-director-maint, rlondhe, sbaker, shardy, srevivo, zbitter
Target Milestone: zstreamKeywords: TestOnly, Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-aodh-3.0.4-3.el7ost Doc Type: Bug Fix
Doc Text:
Prior to this update, when an alarm URL had the trust+http:// scheme, Aodh returned the internally-created trust ID as part of the URL when displaying the alarm data. However, for security reasons, URLs that contain trust IDs are no longer supported. As a result, a read-modify-write of an Aodh alarm with a URL that contained a trust+http alarm failed because Aodh rejected the unchanged alarm URL. With this update, the alarm URL no longer contains the trust ID when displaying an alarm and read-modify-write operations on alarms are successful.
Story Points: ---
Clone Of:
: 1573339 (view as bug list) Environment:
Last Closed: 2019-03-25 10:35:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1573339, 1573340    

Description Petersingh Anburaj 2018-03-20 09:26:46 UTC
Description of problem:

When I try to suspend a stack that has alarm in it, it gives me below error.
~~~
2018-03-19 12:03:14Z [Stack02.cpu_alarm_high]: SUSPEND_FAILED  BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
2018-03-19 12:03:14Z [Stack02]: SUSPEND_FAILED  Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
~~~


Version-Release number of selected component (if applicable):

[stack@instack ~]$ rpm -qa | grep heat
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-common-7.0.6-1.el7ost.noarch
openstack-heat-api-7.0.6-1.el7ost.noarch
openstack-tripleo-heat-templates-5.3.3-1.el7ost.noarch
puppet-heat-9.5.0-2.el7ost.noarch
python-heatclient-1.5.2-1.el7ost.noarch
openstack-heat-engine-7.0.6-1.el7ost.noarch
openstack-heat-templates-0-0.14.1e6015dgit.el7ost.noarch
python-heat-agent-0-0.14.1e6015dgit.el7ost.noarch
openstack-heat-api-cfn-7.0.6-1.el7ost.noarch


How reproducible:
100%


Steps to Reproduce:

1. Create a auto-scaling stack with alarms.
Snippet of template:
~~~
..
..
    cpu_alarm_high:
        type: OS::Aodh::GnocchiAggregationByResourcesAlarm
        properties:
            description: Scale-up if the average CPU > x% for y minute
            metric: cpu_util
            aggregation_method: mean
            granularity: { get_param: time_range }
            evaluation_periods: { get_param: cpu_up_period_time }
            threshold: { get_param: max_cpu }
            resource_type: instance
            comparison_operator: gt
            alarm_actions:
                - str_replace:
                    template: trust+url
                    params:
                        url: {get_attr: [server_scaleup_policy, signal_url]}
            query:
              list_join:
                - ''
                - - {'=': {server_group: {get_param: "OS::stack_id"}}}
..
..
~~~


2. Suspend the stack using the below command.
~~~
openstack stack suspend Stack02
~~~


3. Stack suspend fails with below error.

[stack@instack ~]$ openstack stack list
+--------------------------------------+------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+-----------------+----------------------+--------------+
| eb15d973-09ce-4af8-8c74-9b86e8ba75df | Stack02    | SUSPEND_FAILED  | 2018-03-19T11:30:37Z | None         |
| db0fff87-0a0a-41fa-9466-6a948dcefabd | Stack01    | RESUME_COMPLETE | 2018-03-19T07:44:44Z | None         |
+--------------------------------------+------------+-----------------+----------------------+--------------+


[stack@instack ~]$ openstack stack show Stack02
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                                                     |
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                    | eb15d973-09ce-4af8-8c74-9b86e8ba75df                                                                                                                                      |
| stack_name            | Stack02                                                                                                                                                                   |
| description           | Simple template to deploy a single compute instance                                                                                                                       |
| creation_time         | 2018-03-19T11:30:37Z                                                                                                                                                      |
| updated_time          | None                                                                                                                                                                      |
| stack_status          | SUSPEND_FAILED                                                                                                                                                            |
| stack_status_reason   | Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)     |
| parameters            | OS::project_id: a10de11a32df479d90df59dbadfb5433                                                                                                                          |
..
..


[stack@instack ~]$ openstack stack event list Stack02
2018-03-19 11:30:38Z [Stack02]: CREATE_IN_PROGRESS  Stack CREATE started
2018-03-19 11:30:38Z [Stack02.lb]: CREATE_IN_PROGRESS  state changed
2018-03-19 11:30:40Z [Stack02.lb]: CREATE_COMPLETE  state changed
..
..
2018-03-19 11:31:29Z [Stack02]: CREATE_COMPLETE  Stack CREATE completed successfully
2018-03-19 12:03:14Z [Stack02]: SUSPEND_IN_PROGRESS  Stack SUSPEND started
2018-03-19 12:03:14Z [Stack02.monitor]: SUSPEND_IN_PROGRESS  state changed
2018-03-19 12:03:14Z [Stack02.monitor]: SUSPEND_COMPLETE  state changed
2018-03-19 12:03:14Z [Stack02.cpu_alarm_high]: SUSPEND_IN_PROGRESS  state changed
2018-03-19 12:03:14Z [Stack02.cpu_alarm_high]: SUSPEND_FAILED  BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
2018-03-19 12:03:14Z [Stack02]: SUSPEND_FAILED  Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)


[root@overcloud-controller-0 ~]# tail /var/log/heat/heat-engine.log -n25
2018-03-19 12:03:14.050 402232 INFO heat.engine.resource [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] suspending HealthMonitor "monitor" [364b819d-36ec-4566-928a-291e1c842ea0] Stack "Stack02" [eb15d973-09ce-4af8-8c74-9b86e8ba75df]
2018-03-19 12:03:14.101 402232 INFO heat.engine.resource [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] suspending AodhGnocchiAggregationByResourcesAlarm "cpu_alarm_high" [497fce25-9179-4b21-a833-f735a1a093fc] Stack "Stack02" [eb15d973-09ce-4af8-8c74-9b86e8ba75df]
2018-03-19 12:03:14.938 402232 INFO heat.engine.resource [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] SUSPEND: AodhGnocchiAggregationByResourcesAlarm "cpu_alarm_high" [497fce25-9179-4b21-a833-f735a1a093fc] Stack "Stack02" [eb15d973-09ce-4af8-8c74-9b86e8ba75df]
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource Traceback (most recent call last):
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 770, in _action_recorder
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     yield
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 872, in _do_action
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 336, in wrapper
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     step = next(subtask)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 817, in action_handler_task
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     handler_data = handler(*args)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/alarm_base.py", line 202, in handle_suspend
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     alarm_update)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/aodhclient/v2/alarm.py", line 156, in update
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     data=jsonutils.dumps(alarm)).json()
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/aodhclient/v2/base.py", line 45, in _put
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     return self.client.api.put(*args, **kwargs)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 196, in put
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     return self.request(url, 'PUT', **kwargs)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/aodhclient/client.py", line 38, in request
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource     raise exceptions.from_response(resp, url, method)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource BadRequest: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)
2018-03-19 12:03:14.938 402232 ERROR heat.engine.resource 
2018-03-19 12:03:14.993 402232 INFO heat.engine.stack [req-1b9c23b5-0ddf-42ed-96c5-e4ea78fb4821 ef7d6e1c5b2c49ae9b31e6976641472d a10de11a32df479d90df59dbadfb5433 - - -] Stack SUSPEND FAILED (Stack02): Resource SUSPEND failed: BadRequest: resources.cpu_alarm_high: trust URL cannot contain a trust ID. (HTTP 400) (Request-ID: req-d8a1af95-6176-40e4-81f4-39455ca3476e)


Actual results:
Suspend fails for stack with alarm all the time.


Expected results:
Suspend should be successful.

Comment 2 Zane Bitter 2018-03-22 14:06:56 UTC
It looks a lot like this upstream bug: https://bugs.launchpad.net/aodh/+bug/1737186

Comment 3 Zane Bitter 2018-04-24 20:20:53 UTC
Fix is merged in upstream master (Rocky) and stable/queens.

Comment 6 AMOL LONARE 2018-09-24 05:21:32 UTC
Hello Team,
 Do we have merged packages available for the customer?
 This case is long pending one, we request you to please prioritize this BZ.

Regards,
Amol Lonare

Comment 33 Amit Ugol 2019-03-23 17:06:12 UTC
LGTM

Comment 34 Lon Hohberger 2019-03-25 10:35:42 UTC
According to our records, this should be resolved by openstack-aodh-3.0.4-3.el7ost.  This build is available now.