Bug 1370242 - Keystone overwhelms Ceilometer with Identity Events
Summary: Keystone overwhelms Ceilometer with Identity Events
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 11.0 (Ocata)
Assignee: Pradeep Kilambi
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-25 17:11 UTC by david.costakos
Modified: 2020-03-26 02:54 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-18 20:24:07 UTC
Target Upstream Version:
Embargoed:
david.costakos: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1627094 0 None None None 2016-09-23 16:06:57 UTC

Description david.costakos 2016-08-25 17:11:59 UTC
Description of problem:
When configuring OpenStack from OSP director, keystone is enabled to produce ceilometer events.  These events spam Ceilometer, and any CloudForms instance managing the Overcloud with "identity.authenticate" events.  These events cause unneeded processing on CloudForms and unneeded data storage in ceilometer as they have no practical use


Version-Release number of selected component (if applicable):
openstack-keystone-8.0.1-1.el7ost.noarch
python-tripleoclient-0.3.4-6.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy Overcloud with ceilometer Events
parameter_defaults:
  CeilometerStoreEvents: true
2. login to controller
3. sudo openstack-config --get /etc/keystone/keystone.conf DEFAULT notification driver
messagin

Actual results:
literally nearly 100,000 identity events get created per day.  Here is a sample of about 22 hours from an unused Cloud.

 grep /ManageIQ/System/Event/EmsEvent/OPENSTACK evm.log  | awk '{ print $10 }' | sort | uniq -c
...
 86317 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.authenticate]
   473 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.created.role_assignment]
     2 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.domain.created]
    54 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.endpoint.created]
    23 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.OS-TRUST:trust.created]
    21 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.OS-TRUST:trust.deleted]
    29 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.project.created]
    21 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.project.deleted]
     2 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.region.created]
   473 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.role_assignment.created]
     8 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.role.created]
    18 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.service.created]
   467 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.user.created]
   404 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.user.deleted]
...

NOTE: the 86,317 identity.authenticate events produced by the Overcloud

Expected results:

identity events need not be produced by default for CloudForms to do its thing.  these are essentially SPAM events that use valuable resources

Additional info:

Suggest setting notification_driver to either log or noop in /etc/keystone/keystone.conf

Comment 3 Adam Young 2016-09-07 18:42:46 UTC
This "works as designed" as these events are necssary for any auditing.  They are not SPAM.  Each of these has security implications.  The error seems to be on the CloudForms side in trying to process events it should be ignoring.

This is not configurable in Keystone.

Do you really think that Keystone needs to change here?

Comment 4 Adam Young 2016-09-07 22:20:49 UTC
Note that you can reduce the number events produced by Keystone using a configuration option.  According to the Keystone documentation, 

http://git.openstack.org/cgit/openstack/keystone/tree/etc/keystone.conf.sample?h=9.0.2#n105

[DEFAULT]
notification_opt_out=identity.user.create
notification_opt_out=identity.authenticate.success

This would prevent keystone from sending notifications out on successful authentications or on user creates.

Comment 5 Adam Young 2016-09-23 20:30:34 UTC
According to http://docs.openstack.org/developer/ceilometer/events.html#events-from-notifications

"The general philosophy of notifications in OpenStack is to emit any and all data someone might need, and let the consumer filter out what they are not interested in."


According to http://docs.openstack.org/developer/ceilometer/events.html#converting-notifications-to-events

"the conversion from Notifications to Events is driven by a configuration file (specified by the flag definitions_cfg_file in ceilometer.conf)."

In the overcloud, you can find the specified file at:

/etc/ceilometer/event_definitions.yaml

This has a stanza that starts like this:

- event_type: identity.authenticate

Removing that stanza should prevent Ceilometer from recording the event.

Comment 8 Greg Blomquist 2016-10-28 22:41:58 UTC
This is likely fixed by https://bugzilla.redhat.com/show_bug.cgi?id=1389848


Note You need to log in before you can comment on or make changes to this bug.