Bug 1353519 - gnocchi_aggregation_by_metrics_threshold throws traceback when overlap is not satisfied
Summary: gnocchi_aggregation_by_metrics_threshold throws traceback when overlap is not...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-aodh
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Mehdi ABAAKOUK
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On: 1360399
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-07 11:34 UTC by Yurii Prokulevych
Modified: 2016-08-15 07:18 UTC (History)
7 users (show)

Fixed In Version: openstack-aodh-2.0.3-2.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, when creating an alarm of type `gnocchi_aggregation_by_metrics_threshold`, the evaluator would throw an error, causing an exception. This update addresses this issue by setting `needed_overlap` to always apply on aggregation.
Clone Of:
Environment:
Last Closed: 2016-08-15 07:18:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 338896 0 None MERGED gnocchi: always set needed_overlap for aggregation 2020-11-12 09:11:40 UTC
OpenStack gerrit 338901 0 None MERGED gnocchi: always set needed_overlap for aggregation 2020-11-12 09:11:41 UTC
Red Hat Product Errata RHEA-2016:1597 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 Release Candidate Advisory 2016-08-11 16:06:52 UTC

Description Yurii Prokulevych 2016-07-07 11:34:49 UTC
Description of problem:
When one of metrics that composes gnocchi_aggregation_by_metrics_threshold alarm, stops being updated/is not updated, aodh-evaluator throws exception:

2016-07-07 11:23:31.912 18519 DEBUG aodh.evaluator [-] evaluating alarm bb643280-7c3a-4eea-b9bd-f4369faf6dd8 _evaluate_alarm /usr/lib/python2.7/site-packages/aodh/evaluator/__init__.py:220
2016-07-07 11:23:31.912 18519 DEBUG aodh.evaluator.threshold [-] query stats from 2016-07-07 11:19:31.912641 to 2016-07-07 11:23:31.912641 _bound_duration /usr/lib/python2.7/site-packages/aodh/evaluator/threshol
d.py:79
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi [-] alarm stats retrieval failed
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi Traceback (most recent call last):
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi   File "/usr/lib/python2.7/site-packages/aodh/evaluator/gnocchi.py", line 79, in _statistics
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi     aggregation=rule['aggregation_method'])
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/metric.py", line 229, in aggregation
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi     params=params).json()
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi     return self.client.api.get(*args, **kwargs)
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 173, in get
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi     return self.request(url, 'GET', **kwargs)
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi   File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line 38, in request
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi     raise exceptions.from_response(resp, method)
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi BadRequest: One of the metrics being aggregated doesn't have matching granularity: Metrics 5c069bcc-13b9-429c-aedd-886c64893157, 8bb12f7f-87f2-41d7-ae38
-b053307e3061 can't be aggregated: Less than 100.000000% of datapoints overlap in this timespan (85.71%) (HTTP 400)
2016-07-07 11:23:33.375 18519 ERROR aodh.evaluator.gnocchi 
2016-07-07 11:23:33.376 18519 DEBUG aodh.evaluator.gnocchi [-] sanitize stats [] _sanitize /usr/lib/python2.7/site-packages/aodh/evaluator/gnocchi.py:52
2016-07-07 11:23:33.376 18519 DEBUG aodh.evaluator.gnocchi [-] pruned statistics to 0 _sanitize /usr/lib/python2.7/site-packages/aodh/evaluator/gnocchi.py:56

This has to be handled gracefully. 

Version-Release number of selected component (if applicable):
openstack-aodh-notifier-2.0.1-3.el7ost.noarch
python-aodh-2.0.1-3.el7ost.noarch
openstack-aodh-listener-2.0.1-3.el7ost.noarch
openstack-aodh-common-2.0.1-3.el7ost.noarch
openstack-aodh-evaluator-2.0.1-3.el7ost.noarch
python-aodhclient-0.5.0-1.el7ost.noarch
openstack-aodh-api-2.0.1-3.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. Create alarm:
aodh --debug alarm create \
--type gnocchi_aggregation_by_metrics_threshold \
--name 'AClient-Gn-Aggr-By-Metrics' \
--description 'AClient Gnocchi-Aggregation-By-Metrics-Threshold' \
--severity moderate \
--enabled True \
--alarm-action 'log://' \
--ok-action 'log://' \
--insufficient-data-action 'log://' \
--comparison-operator ge \
--evaluation-periods 3 \
--threshold 4.0 \
--granularity 60 \
--aggregation-method mean \
--metrics 5c069bcc-13b9-429c-aedd-886c64893157 \
--metrics 8bb12f7f-87f2-41d7-ae38-b053307e3061

"""  5c069bcc-13b9-429c-aedd-886c64893157 is "radosgw.api.request" on ceph-alarm-2 resource
""" 8bb12f7f-87f2-41d7-ae38-b053307e3061 is "radosgw.api.request" on ceph-alarm-3 resource

Comment 3 Yurii Prokulevych 2016-07-29 14:33:43 UTC
Alarm transitioned between states after one metric stopped being updated.

Packages:
---------
openstack-aodh-notifier-2.0.3-2.el7ost.noarch
python-aodh-2.0.3-2.el7ost.noarch
openstack-aodh-evaluator-2.0.3-2.el7ost.noarch
openstack-aodh-common-2.0.3-2.el7ost.noarch
openstack-aodh-listener-2.0.3-2.el7ost.noarch
openstack-aodh-api-2.0.3-2.el7ost.noarch

Comment 5 errata-xmlrpc 2016-08-15 07:18:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html


Note You need to log in before you can comment on or make changes to this bug.