Description of problem: In RHOS-10 w/ disabled Telemetry we're pushing the noop notification driver to all controllers, for all services with oslo_messaging_notifications config SECTION: [root@controller-01 ~]# for i in $(grep -ri oslo_messaging_notifications /etc/* 2>/dev/null |awk -F: '{print $1}'); do printf "%-40s" $i; crudini --get $i oslo_messaging_notifications driver; done &> ~/oslo_messaging_notifications_$HOSTNAME.txt cat os [root@controller-01 ~]# [root@controller-01 ~]# cat oslo_messaging_notifications_controller-01.openstack.local.txt | grep "conf " /etc/aodh/aodh.conf noop /etc/ceilometer/ceilometer.conf noop /etc/cinder/cinder.conf noop /etc/cinder/cinder-cesano.conf noop /etc/cinder/cinder-rozzano.conf noop /etc/glance/glance-registry.conf noop /etc/glance/glance-api.conf noop /etc/heat/heat.conf noop /etc/keystone/keystone.conf noop /etc/manila/manila.conf noop /etc/neutron/neutron.conf noop /etc/nova/nova.conf noop /etc/sahara/sahara.conf noop [root@controller-02 ~]# for i in $(grep -ri oslo_messaging_notifications /etc/* 2>/dev/null |awk -F: '{print $1}'); do printf "%-40s" $i; crudini --get $i oslo_messaging_notifications driver; done &> ~/oslo_messaging_notifications_$HOSTNAME.txt [root@controller-02 ~]# cat oslo_messaging_notifications_controller-02.openstack.local.txt | grep "conf " /etc/aodh/aodh.conf noop /etc/ceilometer/ceilometer.conf noop /etc/cinder/cinder.conf noop /etc/cinder/cinder-cesano.conf noop /etc/cinder/cinder-rozzano.conf noop /etc/glance/glance-registry.conf noop /etc/glance/glance-api.conf noop /etc/heat/heat.conf noop /etc/keystone/keystone.conf noop /etc/manila/manila.conf noop /etc/neutron/neutron.conf noop /etc/nova/nova.conf noop /etc/sahara/sahara.conf noop [root@controller-03 ~]# for i in $(grep -ri oslo_messaging_notifications /etc/* 2>/dev/null |awk -F: '{print $1}'); do printf "%-40s" $i; crudini --get $i oslo_messaging_notifications driver; done &> ~/oslo_messaging_notifications_$HOSTNAME.txt [root@controller-03 ~]# cat oslo_messaging_notifications_controller-03.openstack.local.txt | grep "conf " /etc/aodh/aodh.conf noop /etc/ceilometer/ceilometer.conf noop /etc/cinder/cinder.conf noop /etc/cinder/cinder-cesano.conf noop /etc/cinder/cinder-rozzano.conf noop /etc/glance/glance-registry.conf noop /etc/glance/glance-api.conf noop /etc/heat/heat.conf noop /etc/keystone/keystone.conf noop /etc/manila/manila.conf noop /etc/neutron/neutron.conf noop /etc/nova/nova.conf noop /etc/sahara/sahara.conf noop << ALL Controller nodes get rebooted. >> [root@controller-03 ~]# rabbitmqctl purge_queue notifications.info Purging queue 'notifications.info' in vhost '/' ... [root@controller-03 ~]# rabbitmqctl list_queues | grep notifications notifications.error 0 notifications.warn 0 notifications.debug 0 notifications.critical 0 notifications.sample 0 notifications.audit 0 versioned_notifications.error 0 versioned_notifications.info 0 notifications.info 0 But after 10 minutes: [root@controller-03 ~]# rabbitmqctl list_queues | grep notifications notifications.error 0 notifications.warn 0 notifications.debug 0 notifications.critical 0 notifications.sample 0 notifications.audit 0 versioned_notifications.error 0 versioned_notifications.info 0 notifications.info 68 >> notifications.info still grows...
Not sure what we can do from the RabbitMQ side. Someone should stop either creating these notifications.info queues or better stop pushing data into it. Perhaps it should be reassigned to oslo-messaging? Also I can see sign of a bug 1436784, but it mostly a cosmetic and does not affect cluster in this particular case.
Switching to oslo as per Peter's recommendation.
"In RHOS-10 w/ disabled Telemetry we're pushing the noop notification driver to all controllers, for all services with oslo_messaging_notifications config SECTION: (...)" Did you follow the following documentation to disable Telemetry? https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/disable_telemetry.html It suggests to use this command: "openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml" If you disabled notifications manually, are you sure that you did not forget a service? Maybe there is one remaining service which still sends notifications?
We are closing this issue because we do not have all of the information required to make further progress. Please re-open if you are able to provide the required information.
The problem is present in RHOSP 10 and later: args='/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log'.split(' ') from oslo_config import cfg from neutron.common import config from neutron.common import rpc as n_rpc config.init(args[2:]) for k, v in cfg.CONF.oslo_messaging_notifications.iteritems(): print k, v transport_url None topics ['notifications'] driver ['neutron.openstack.common.notifier.rpc_notifier', 'noop'] The problem is originating in /usr/share/neutron/neutron-dist.conf and commenting the notification_driver solves the problem: [DEFAULT] verbose = True lock_path = $state_path/lock #notification_driver = neutron.openstack.common.notifier.rpc_notifier allow_overlapping_ips = True use_stderr = False api_paste_config = /usr/share/neutron/api-paste.ini [agent] root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
When commenting notification_driver in /usr/share/neutron/neutron-dist.conf, the notification driver is properly set to noop: transport_url None topics ['notifications'] driver ['noop']
And with heat.conf, the problem is that even by setting heat::notification_driver in the environment file (network-environment.yaml ) , it never lands in /etc/heat/heat.conf. I guess we have 2 bugs here . Should I open another one or clone this one ?
Oh wow, I didn't know that the option accepted multiple values :-) I guess that there is an use case to have multiple notifier drivers. But here, we want to not send notifications, so we must have exactly one driver which is: "noop". I'm not sure which option is the best to get exactly one driver. https://github.com/openstack/tripleo-heat-templates/blob/master/environments/disable-telemetry.yaml In the YAML, I see a single value: parameter_defaults: NotificationDriver: 'noop'
Ack, so I think we're all in agreement that we should just remove it from neutron-dist.conf. Thanks, everyone *** This bug has been marked as a duplicate of bug 1601593 ***