Bug 1212158

Summary: notifications are not being sent by OpenStack services because notification_driver is not defined in their config files
Product: Red Hat OpenStack Reporter: Richard Su <rwsu>
Component: instack-undercloudAssignee: Ben Nemec <bnemec>
Status: CLOSED ERRATA QA Contact: Amit Ugol <augol>
Severity: high Docs Contact:
Priority: urgent    
Version: 7.0 (Kilo)CC: augol, david.costakos, dh3, dmacpher, hbrock, jason.dobies, jcoufal, jhardy, jslagle, kbasil, kmorey, mberube, mburns, mcornea, nstephan, ohochman, rhel-osp-director-maint, rrasouli, tzumainn
Target Milestone: gaKeywords: ZStream
Target Release: 8.0 (Liberty)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: instack-undercloud-2.2.6-1.el7ost Doc Type: Enhancement
Doc Text:
This updates provides OpenStack notifications. Previously there were external consumers of OpenStack notifications that could not interface with director-deployed cloud because notifications were not enabled. Now director enables notifications for external consumers.
Story Points: ---
Clone Of:
: 1225163 (view as bug list) Environment:
Last Closed: 2016-04-07 21:37:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1225163    

Description Richard Su 2015-04-15 16:54:12 UTC
Description of problem:

Notifications (https://wiki.openstack.org/wiki/SystemUsageData) sent to the notifications.info queue which are consumed by Ceilometer or ManageIQ are not being sent by OpenStack services. The services configuration file are leaving the notification_driver undefined. I believe this is a problem on both the undercloud and overcloud.

For nova and heat, I had to manually change the config files and restart services to start getting notifications. 

/etc/heat/heat.conf
notification_driver=glance.openstack.common.notifier.rpc_notifier
notification_topics=notifications

/etc/nova/nova.conf
notification_driver=messaging
notification_topics=notifications

Ideally, puppet would set the appropriate notification_driver and any other settings for all services.

There might be additional work needed upstream because not every puppet module allows you to set the notification driver. For example nova does, but heat and ironic does not.


Version-Release number of selected component (if applicable):
instack-0.0.6.4-g57c723a.el7.centos.noarch
instack-undercloud-2.0.0-dev1606.el7.centos.noarch
openstack-tripleo-puppet-elements-0.0.0-post45.el7.centos.noarch

How reproducible:
always

Steps to Reproduce:
1. Deploy instack undercloud
2. Observe notification.info queue, from RabbitMQ management UI. No messages flow into the queue when you deploy the overcloud.

Actual results:
Notifications are not sent

Expected results:
Notifications are sent

Additional info:

Comment 3 Richard Su 2015-04-20 23:32:11 UTC
Correction.

The heat configuration file should have the following lines.

/etc/heat/heat.conf
notification_driver=messaging
notification_topics=notifications

Comment 8 Jaromir Coufal 2015-12-08 09:55:37 UTC
Urgent for OSP8, not a blocker (removing a flag).

Comment 10 Marco Berube 2016-03-10 17:06:10 UTC
I believe "notification_driver=messaging" needs to be added to more than just nova and heat configuration file.

When adding "notification_driver=messaging" only in Nova, I am not getting all instances events in Cloudforms.  As an example, Cloudforms will get messages about an instance power off/on.  But Cloudforms will not see a new instances booting up.  

I have added "notification_driver=messaging" to all configuration files: nova.conf, neutron.conf, heat.conf, keystone.conf, glance.conf, etc... And now Cloudforms can see all messages, including newly created instanced directly from Openstack.

Somebody should verify exactly what list of messages are required by Cloudforms or make sure all services have this setting (unless this is causing an issue).

Comment 11 Tzu-Mainn Chen 2016-03-14 20:29:31 UTC
The related patch adds the relevant configuration for Keystone, Glance, Nova, Heat, and Neutron, so I think we're okay here!

Comment 13 Ronnie Rasouli 2016-03-24 12:59:02 UTC
See below output all relevant components configured with messaging
/etc/glance/glance-api.conf:508:notification_driver =messaging
/etc/heat/heat.conf:401:notification_driver =messaging
/etc/keystone/keystone.conf:238:notification_driver =messaging
/etc/neutron/neutron.conf:590:notification_driver=messaging
/etc/neutron/neutron.conf:1123:notification_drivers = message_queue
/etc/nova/nova.conf:1485:notification_driver =messaging
/etc/puppet/hieradata/puppet-stack-config.yaml:223:keystone::notification_driver: messaging
/etc/puppet/hieradata/puppet-stack-config.yaml:316:nova::notification_driver: messaging

Comment 14 Eoghan Glynn 2016-03-31 12:47:56 UTC
*** Bug 1314616 has been marked as a duplicate of this bug. ***

Comment 16 errata-xmlrpc 2016-04-07 21:37:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html