Bug 1258898

Summary: [RFE] Remove pipeline logic from ceilometer API service
Product: Red Hat OpenStack Reporter: Eoghan Glynn <eglynn>
Component: openstack-ceilometerAssignee: Eoghan Glynn <eglynn>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: medium Docs Contact:
Priority: low    
Version: 8.0 (Liberty)CC: eglynn, jruzicka, jschluet, nbarcet, yeylon
Target Milestone: betaKeywords: FutureFeature, OtherQA
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
URL: https://blueprints.launchpad.net/ceilometer/+spec/api-no-pipeline
Whiteboard: upstream_milestone_liberty-2 upstream_definition_approved upstream_status_implemented
Fixed In Version: openstack-ceilometer-5.0.0-1.el7ost Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-07 21:06:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eoghan Glynn 2015-09-01 13:28:29 UTC
Currently the ceilometer-api service uses the pipeline logic in a fashion similar to the agents when processing samples POSTed via the REST API. This introduces unnecessary complexity in code and testing/deployment.

Instead the need for the pipeline at the API layer could be avoided with the simple expedient of emitting notifcations containing POSTed samples for the ceilometer notification agent to consume and dispatch to the collector in the normal way.

Comment 3 Chris Dent 2015-09-09 14:34:58 UTC
The primary thing that needs to be confirmed when testing this change is that POST-ing samples still works correctly (the samples are stored) using the new code. A secondary thing to confirm is that POST-ed samples that should be transformed by the pipeline are. We can do this by creating some samples using python-ceilometerclient:

This creates an arbitrary meter:

$ ceilometer sample-create -r $(uuidgen) -m cow --meter-type gauge --meter-unit C --sample-volume 5

we can confirm it was received by looking at:

$ ceilometer sample-list -q meter=cow
+--------------------------------------+--------------------------------------+------+-------+--------+------+----------------------------+
| ID                                   | Resource ID                          | Name | Type  | Volume | Unit | Timestamp                  |
+--------------------------------------+--------------------------------------+------+-------+--------+------+----------------------------+
| 62848c20-56fc-11e5-bd04-3417ebd4f75d | f95be10e-e9f0-46aa-b6ad-2c3ff6c87297 | cow  | gauge | 5.0    | C    | 2015-09-09T14:09:28.926000 |
+--------------------------------------+--------------------------------------+------+-------+--------+------+----------------------------+
 
Then to confirm that the samples are going through the notification agent, shut down the notification agent and create a sample (as above), confirm that it is _not_ there (as above). Then restart the notification agent, wait a few minutes and confirm the sample (which will have been waiting in the message bus) is saved:

$ ceilometer sample-list -q meter=cow
+--------------------------------------+--------------------------------------+------+-------+--------+------+----------------------------+
| ID                                   | Resource ID                          | Name | Type  | Volume | Unit | Timestamp                  |
+--------------------------------------+--------------------------------------+------+-------+--------+------+----------------------------+
| 1ffbd8b6-56fe-11e5-bd04-3417ebd4f75d | 12494bf6-df3e-41f4-9585-ec94b2eae053 | cow  | gauge | 10.0   | C    | 2015-09-09T14:21:56.294000 |
| 62848c20-56fc-11e5-bd04-3417ebd4f75d | f95be10e-e9f0-46aa-b6ad-2c3ff6c87297 | cow  | gauge | 5.0    | C    | 2015-09-09T14:09:28.926000 |
+--------------------------------------+--------------------------------------+------+-------+--------+------+----------------------------+


Finally we can confirm the operation of the pipeline by creating samples using that we know will resort in a transform. The default pipeline.yaml will transform 'cpu' samples into 'cpu_util'. In order for this test to work cleanly, no existing instances should be present:

$ export INSTANCE=$(uuidgen) 
$ for i in 5000 10000 15000 20000 ; do ceilometer sample-create -r $INSTANCE -m cpu --meter-type cumulative --meter-unit ns --sample-volume $i ; done
$ ceilometer sample-list -q meter=cpu_util

In the log file for the notification agent (when debugging in turned on) you should see entries like:

2015-09-09 14:32:22.778 25382 DEBUG ceilometer.transformer.conversions [req-77a78b1c-1314-4eea-858d-cac4f8c072e6 - - - - -] handling sample <name: cpu, volume: 15000.0, resource_id: 87a42ab8-e8ad-4178-9800-4fb740db7ae4, timestamp: 2015-09-09T14:32:22.774625> handle_sample /home/opt/stack/ceilometer/ceilometer/transformer/conversions.py:115

where the resource_id matches the value of $INSTANCE

Comment 5 Yurii Prokulevych 2016-01-14 07:33:58 UTC
Verified according test plan.

Comment 7 errata-xmlrpc 2016-04-07 21:06:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html