Bug 1294463 - error notifications projects recording (specifically nova) send notifications on error topic when an error occurs
error notifications projects recording (specifically nova) send notification...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer (Show other bugs)
6.0 (Juno)
Unspecified Unspecified
unspecified Severity medium
: ---
: 8.0 (Liberty)
Assigned To: Mehdi ABAAKOUK
Yurii Prokulevych
: Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-28 05:26 EST by Pablo Caruana
Modified: 2016-11-14 14:56 EST (History)
11 users (show)

See Also:
Fixed In Version: openstack-ceilometer-5.0.5-1.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-14 14:56:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1364708 None None None 2015-12-28 05:26 EST
OpenStack gerrit 349838 None None None 2016-09-12 03:19 EDT

  None (edit)
Description Pablo Caruana 2015-12-28 05:26:04 EST
Description of problem:

 start recording error notifications

    projects (specifically nova) send notifications on error topic when
    an error occurs. we should capture this in events.

Discussion at https://review.openstack.org/#/c/152525/

The following ones were tried during OSP 7 kilo cycle (looked and even applied the diffs for) 
https://review.openstack.org/#/c/153362/, \https://review.openstack.org/#/c/152525/
 but it turns out we do not have the required code on  OSP 6 Juno in e.g. /usr/lib/python2.7/site-packages/ceilometer/ to deal with it.  I've since reverted those diffs.

rabbitmq know of no consumers to notifications.error, e.g:

[root@lab ~]# rabbitmqctl list_queues | grep notifications
notifications.error	100
notifications.info	0

[root@mlab ~]# rabbitmqctl list_consumers | grep notifications.info | wc -l
225
[root@mlab ~]# rabbitmqctl list_consumers | grep notifications.error | wc -l
0
Comment 8 Julien Danjou 2016-07-13 08:39:43 EDT
The problem has nothing to do with Ceilometer unfortunately. As soon as you enable notifications in any OpenStack project, and if that projects emits an "error" notification, it ends up into that queue.

No project in OpenStack consume that queue. Ceilometer neither. So it might en up being big for nothing.

There should probably be a way to disable this queue, or to make Ceilometer consume it eventually.

I'm gonna reassign to oslo.messaging as this is likely the place where such (dis|)enablements should be done.
Comment 9 John Eckersberg 2016-08-26 13:09:28 EDT
(In reply to Julien Danjou from comment #8)
> The problem has nothing to do with Ceilometer unfortunately. As soon as you
> enable notifications in any OpenStack project, and if that projects emits an
> "error" notification, it ends up into that queue.
> 
> No project in OpenStack consume that queue. Ceilometer neither. So it might
> en up being big for nothing.
> 
> There should probably be a way to disable this queue, or to make Ceilometer
> consume it eventually.
> 
> I'm gonna reassign to oslo.messaging as this is likely the place where such
> (dis|)enablements should be done.

To disable emitting notifications to the unconsumed error queue, you can set on OSP6:

notification_driver = noop

In the configuration for nova (and others).

Does that resolve this issue?
Comment 10 Mehdi ABAAKOUK 2016-09-12 03:23:01 EDT
Changing notification_driver to noop will also disable the notification.info queue. The issue is about other notification queues (error/audit/...). 

Ceilometer upstream decides to consume all notifications queues with Ceilometer since they are no way to just consume one queue and disable other queues without breaking oslo.messaging API/contract.
Comment 16 Mehdi ABAAKOUK 2016-11-07 08:46:47 EST
Thanks
Comment 19 errata-xmlrpc 2016-11-14 14:56:02 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2714.html

Note You need to log in before you can comment on or make changes to this bug.