Bug 986410
| Summary: | [RFE] Aggregation over multiple alarm states | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eoghan Glynn <eglynn> |
| Component: | openstack-ceilometer | Assignee: | Eoghan Glynn <eglynn> |
| Status: | CLOSED ERRATA | QA Contact: | Kevin Whitney <kwhitney> |
| Severity: | low | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.0 | CC: | ajeain, eglynn, jruzicka, mlopes, pbrady, sgordon, sradvan, srevivo |
| Target Milestone: | rc | Keywords: | FutureFeature, OtherQA |
| Target Release: | 4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | https://blueprints.launchpad.net/ceilometer/+spec/alarming-logical-combination | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-ceilometer-2013.2-0.12.rc2.el6ost | Doc Type: | Enhancement |
| Doc Text: |
Feature: Aggregation of the states of multiple basic alarms into overarching meta-alarms.
Reason: Reducing noise from detailed monitoring, also allowing alarm-driven workflows (e.g. Heat autoscaling) to be gated by more complex logical conditions.
Result: Combination alarms may be created to combine the states of the under-pinning alarms via logical AND or OR.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-12-20 00:14:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 988358 | ||
| Bug Blocks: | 973191, 975499, 1055813 | ||
|
Description
Eoghan Glynn
2013-07-19 17:11:45 UTC
Merged upstream as an FFE for Havana RC1: https://github.com/openstack/ceilometer/commit/d30bf2fa2 https://github.com/openstack/ceilometer/commit/985f48270 How To Test
===========
0. Install packstack allinone, then spin up an instance in the usual way.
Ensure the compute agent is gathering metrics at a reasonable cadence (every 60s for example instead of every 10mins as per the default):
sudo sed -i '/^ *name: cpu_pipeline$/ { n ; s/interval: 600$/interval: 60/ }' /etc/ceilometer/pipeline.yaml
sudo service openstack-ceilometer-compute restart
Ensure the ceilometer alarm services are installed and running:
sudo yum install -y openstack-ceilometer-alarm
export CEILO_ALARM_SVCS='evaluator notifier'
for svc in $CEILO_ALARM_SVCS; do sudo service openstack-ceilometer-alarm-$svc start; done
1. Create multiple basic alarms with thresholds sufficiently low that they are guaranteed to go into alarm:
for i in $(seq 5)
do
ceilometer alarm-threshold-create --name basic_cpu_low_threshold_${i} \
--meter-name cpu_util --threshold 0.01 --comparison-operator gt --statistic avg \
--period 60 --evaluation-periods 1 \
--alarm-action 'log://' \
--query resource_id=$INSTANCE_ID
done
2. Create a meta-alarm combining (with logical AND) the state of these basic alarms:
ALARM_IDS=
for a in `ceilometer alarm-list | awk -F\| '/basic_cpu_/ {print $2}'`; do ALARM_IDS="$ALARM_IDS --alarm_ids $a"; done
ceilometer --debug alarm-combination-create --name combination_cpu_low --description 'combination of high CPU util alarms' --alarm-action 'log://' $ALARM_IDS
Ensure that this combination alarm transitions into the alarm state within one evaluation period (60s by default):
sleep 60 ; ceilometer alarm-list | grep combination_cpu_low
3. Create another set of basic alarms with thresholds sufficiently high that they are guaranteed not to go into alarm:
for i in $(seq 5)
do
ceilometer alarm-threshold-create --name basic_cpu_high_threshold_${i} \
--meter-name cpu_util --threshold 99.99 --comparison-operator gt --statistic max \
--period 60 --evaluation-periods 1 \
--alarm-action 'log://' \
--query resource_id=$INSTANCE_ID
done
4. Create 2 further meta-alarms combining (with logical AND & OR respectively) the state of these basic alarms:
ALARM_IDS=
for a in `ceilometer alarm-list | awk -F\| '/basic_cpu_/ {print $2}'`; do ALARM_IDS="$ALARM_IDS --alarm_ids $a"; done
ceilometer --debug alarm-combination-create --name combination_cpu_mixed_and --description 'combination (AND) of mixed CPU util alarms' --alarm-action 'log://' $ALARM_IDS --operator and
ceilometer --debug alarm-combination-create --name combination_cpu_mixed_or --description 'combination (OR) of mixed CPU util alarms' --alarm-action 'log://' $ALARM_IDS --operator or
Ensure that this combination alarms transition into the alarm and ok state (for the combination_cpu_mixed_or and combination_cpu_mixed_and alarms respectively) within one evaluation period (60s by default):
sleep 60 ; ceilometer alarm-list | grep combination_cpu_mixed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html |