Bug 1563389
| Summary: | notifications.info keeps growing even after passing 'driver=noop' | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Irina Petrova <ipetrova> |
| Component: | puppet-oslo | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | nlevinki <nlevinki> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | apevec, chjones, dhill, ipetrova, jeckersb, jjoyce, jschluet, lhh, michele, slinaber, srevivo, tvignaud, vstinner |
| Target Milestone: | --- | Keywords: | Reopened, Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-07-23 13:23:34 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Not sure what we can do from the RabbitMQ side. Someone should stop either creating these notifications.info queues or better stop pushing data into it. Perhaps it should be reassigned to oslo-messaging? Also I can see sign of a bug 1436784, but it mostly a cosmetic and does not affect cluster in this particular case. Switching to oslo as per Peter's recommendation. "In RHOS-10 w/ disabled Telemetry we're pushing the noop notification driver to all controllers, for all services with oslo_messaging_notifications config SECTION: (...)" Did you follow the following documentation to disable Telemetry? https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/disable_telemetry.html It suggests to use this command: "openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml" If you disabled notifications manually, are you sure that you did not forget a service? Maybe there is one remaining service which still sends notifications? We are closing this issue because we do not have all of the information required to make further progress. Please re-open if you are able to provide the required information. The problem is present in RHOSP 10 and later:
args='/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log'.split(' ')
from oslo_config import cfg
from neutron.common import config
from neutron.common import rpc as n_rpc
config.init(args[2:])
for k, v in cfg.CONF.oslo_messaging_notifications.iteritems(): print k, v
transport_url None
topics ['notifications']
driver ['neutron.openstack.common.notifier.rpc_notifier', 'noop']
The problem is originating in /usr/share/neutron/neutron-dist.conf and commenting the notification_driver solves the problem:
[DEFAULT]
verbose = True
lock_path = $state_path/lock
#notification_driver = neutron.openstack.common.notifier.rpc_notifier
allow_overlapping_ips = True
use_stderr = False
api_paste_config = /usr/share/neutron/api-paste.ini
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
When commenting notification_driver in /usr/share/neutron/neutron-dist.conf, the notification driver is properly set to noop: transport_url None topics ['notifications'] driver ['noop'] And with heat.conf, the problem is that even by setting heat::notification_driver in the environment file (network-environment.yaml ) , it never lands in /etc/heat/heat.conf. I guess we have 2 bugs here . Should I open another one or clone this one ? Oh wow, I didn't know that the option accepted multiple values :-) I guess that there is an use case to have multiple notifier drivers. But here, we want to not send notifications, so we must have exactly one driver which is: "noop". I'm not sure which option is the best to get exactly one driver. https://github.com/openstack/tripleo-heat-templates/blob/master/environments/disable-telemetry.yaml In the YAML, I see a single value: parameter_defaults: NotificationDriver: 'noop' Ack, so I think we're all in agreement that we should just remove it from neutron-dist.conf. Thanks, everyone *** This bug has been marked as a duplicate of bug 1601593 *** |
Description of problem: In RHOS-10 w/ disabled Telemetry we're pushing the noop notification driver to all controllers, for all services with oslo_messaging_notifications config SECTION: [root@controller-01 ~]# for i in $(grep -ri oslo_messaging_notifications /etc/* 2>/dev/null |awk -F: '{print $1}'); do printf "%-40s" $i; crudini --get $i oslo_messaging_notifications driver; done &> ~/oslo_messaging_notifications_$HOSTNAME.txt cat os [root@controller-01 ~]# [root@controller-01 ~]# cat oslo_messaging_notifications_controller-01.openstack.local.txt | grep "conf " /etc/aodh/aodh.conf noop /etc/ceilometer/ceilometer.conf noop /etc/cinder/cinder.conf noop /etc/cinder/cinder-cesano.conf noop /etc/cinder/cinder-rozzano.conf noop /etc/glance/glance-registry.conf noop /etc/glance/glance-api.conf noop /etc/heat/heat.conf noop /etc/keystone/keystone.conf noop /etc/manila/manila.conf noop /etc/neutron/neutron.conf noop /etc/nova/nova.conf noop /etc/sahara/sahara.conf noop [root@controller-02 ~]# for i in $(grep -ri oslo_messaging_notifications /etc/* 2>/dev/null |awk -F: '{print $1}'); do printf "%-40s" $i; crudini --get $i oslo_messaging_notifications driver; done &> ~/oslo_messaging_notifications_$HOSTNAME.txt [root@controller-02 ~]# cat oslo_messaging_notifications_controller-02.openstack.local.txt | grep "conf " /etc/aodh/aodh.conf noop /etc/ceilometer/ceilometer.conf noop /etc/cinder/cinder.conf noop /etc/cinder/cinder-cesano.conf noop /etc/cinder/cinder-rozzano.conf noop /etc/glance/glance-registry.conf noop /etc/glance/glance-api.conf noop /etc/heat/heat.conf noop /etc/keystone/keystone.conf noop /etc/manila/manila.conf noop /etc/neutron/neutron.conf noop /etc/nova/nova.conf noop /etc/sahara/sahara.conf noop [root@controller-03 ~]# for i in $(grep -ri oslo_messaging_notifications /etc/* 2>/dev/null |awk -F: '{print $1}'); do printf "%-40s" $i; crudini --get $i oslo_messaging_notifications driver; done &> ~/oslo_messaging_notifications_$HOSTNAME.txt [root@controller-03 ~]# cat oslo_messaging_notifications_controller-03.openstack.local.txt | grep "conf " /etc/aodh/aodh.conf noop /etc/ceilometer/ceilometer.conf noop /etc/cinder/cinder.conf noop /etc/cinder/cinder-cesano.conf noop /etc/cinder/cinder-rozzano.conf noop /etc/glance/glance-registry.conf noop /etc/glance/glance-api.conf noop /etc/heat/heat.conf noop /etc/keystone/keystone.conf noop /etc/manila/manila.conf noop /etc/neutron/neutron.conf noop /etc/nova/nova.conf noop /etc/sahara/sahara.conf noop << ALL Controller nodes get rebooted. >> [root@controller-03 ~]# rabbitmqctl purge_queue notifications.info Purging queue 'notifications.info' in vhost '/' ... [root@controller-03 ~]# rabbitmqctl list_queues | grep notifications notifications.error 0 notifications.warn 0 notifications.debug 0 notifications.critical 0 notifications.sample 0 notifications.audit 0 versioned_notifications.error 0 versioned_notifications.info 0 notifications.info 0 But after 10 minutes: [root@controller-03 ~]# rabbitmqctl list_queues | grep notifications notifications.error 0 notifications.warn 0 notifications.debug 0 notifications.critical 0 notifications.sample 0 notifications.audit 0 versioned_notifications.error 0 versioned_notifications.info 0 notifications.info 68 >> notifications.info still grows...