Bug 1736786

For neutron, the fix is quite simple:

/usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp:

Replace:
  neutron_config {
    'DEFAULT/notification_driver': value => 'messaging';
  }

for:
if str2bool(hiera('enable_telemetry', true)) {
  neutron_config {
    'DEFAULT/notification_driver': value => 'messaging';
  }
} else {
  neutron_config {
    'DEFAULT/notification_driver': value => 'noop';
  }
}


... I tried looking at the other services but they are hardcoded here:

/usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template:heat::notification_driver: 'messaging'
/usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template:keystone::notification_driver: messaging
/usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template:nova::notification_driver: messaging

and the result:
heat/heat.conf:driver =messaging
keystone/keystone.conf:driver =messaging
nova/nova.conf:driver=messaging

/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py:


    cfg.BoolOpt('enable_telemetry',
                default=True,
                help=('Whether to install Telemetry services '
                      '(ceilometer, aodh) in the Undercloud.')
                ),
    cfg.StrOpt('messaging_driver',
                default="messaging",
                help=('Whether to install Telemetry services '
                      '(ceilometer, aodh) in the Undercloud.')
                ),


and:

[root@undercloud-3 /]# grep -r MESSAGING /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template 
heat::notification_driver: '{{MESSAGING_DRIVER}}'
keystone::notification_driver: {{MESSAGING_DRIVER}}
nova::notification_driver: {{MESSAGING_DRIVER}}

But perhaps it would be cleaner to simply remove the notification_driver from the puppet-stack-config.yaml.template and do the same as neutron.

Something likes (didn't test it):

if str2bool(hiera('enable_telemetry', true)) {
  neutron_config {
    'DEFAULT/notification_driver': value => 'messaging';
  }
  heat_config {
    'DEFAULT/notification_driver': value => 'messaging';
  }
  keystone_config {
    'DEFAULT/notification_driver': value => 'messaging';
  }
  nova_config {
    'DEFAULT/notification_driver': value => 'messaging';
  }
} else {
  neutron_config {
    'DEFAULT/notification_driver': value => 'noop';
  }
  heat_config {
    'DEFAULT/notification_driver': value => 'noop';
  }
  keystone_config {
    'DEFAULT/notification_driver': value => 'noop';
  }
  nova_config {
    'DEFAULT/notification_driver': value => 'noop';
  }
}


and remove from /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template the following lines:
heat::notification_driver: 'messaging'
keystone::notification_driver: messaging
nova::notification_driver: messaging

The last comment works:

[root@undercloud-3 etc]# grep driver.*=.*noop */*.conf
heat/heat.conf:notification_driver=noop
keystone/keystone.conf:notification_driver=noop
neutron/neutron.conf:notification_driver=noop
nova/nova.conf:notification_driver=noop

Created attachment 1599481 [details]
Patch to fix enable_telemetry in newton.

I pushed https://review.opendev.org/#/c/674160/ upstream for the rocky branch .

I found this bug in OSP14, as well.  In an environment with "enable_telemetry = true" set in undercloud.conf, the various processes (e.g. heat, keystone, nova, etc.) all get "driver=messagingv2" set in the configuration files. Therefore, they all send messages to notifications.info. But there are processes listening to that queue, so the queue stays empty.

In an environment with "enable_telemetry = false" set in undercloud.conf, those processes still have "driver=messagingv2" set in the configuration files, so they still send messages to notification.info, but there are no consumers on that queue, leading to a situation where the queue gets bigger and bigger forever.


With enable_telemetry = false (this environment had only been running for about 20 minutes, so there are not many messages in the queue):

$ grep telemetry undercloud.conf 
enable_telemetry = false

# grep ^driver=messa /var/lib/config-data/*/etc/*/*.conf
/var/lib/config-data/glance_api/etc/glance/glance-api.conf:driver=messagingv2
/var/lib/config-data/heat_api_cfn/etc/heat/heat.conf:driver=messagingv2
/var/lib/config-data/heat_api/etc/heat/heat.conf:driver=messagingv2
/var/lib/config-data/heat/etc/heat/heat.conf:driver=messagingv2
/var/lib/config-data/keystone/etc/keystone/keystone.conf:driver=messagingv2
/var/lib/config-data/mistral/etc/mistral/mistral.conf:driver=messagingv2
/var/lib/config-data/neutron/etc/neutron/neutron.conf:driver=messagingv2
/var/lib/config-data/nova/etc/nova/nova.conf:driver=messagingv2
/var/lib/config-data/nova_metadata/etc/nova/nova.conf:driver=messagingv2
/var/lib/config-data/nova_placement/etc/nova/nova.conf:driver=messagingv2

# docker exec -it rabbitmq bash
()[rabbitmq@osp14-undercloud /]$ rabbitmqctl list_queues name consumers messages | grep notifications
notifications.info	0	45


With enable_telemetry = true:

$ grep telemetry undercloud.conf 
enable_telemetry = true

# grep ^driver=messa /var/lib/config-data/*/etc/*/*.conf
/var/lib/config-data/aodh/etc/aodh/aodh.conf:driver=messagingv2
/var/lib/config-data/ceilometer/etc/ceilometer/ceilometer.conf:driver=messagingv2
/var/lib/config-data/glance_api/etc/glance/glance-api.conf:driver=messagingv2
/var/lib/config-data/heat_api_cfn/etc/heat/heat.conf:driver=messagingv2
/var/lib/config-data/heat_api/etc/heat/heat.conf:driver=messagingv2
/var/lib/config-data/heat/etc/heat/heat.conf:driver=messagingv2
/var/lib/config-data/keystone/etc/keystone/keystone.conf:driver=messagingv2
/var/lib/config-data/mistral/etc/mistral/mistral.conf:driver=messagingv2
/var/lib/config-data/neutron/etc/neutron/neutron.conf:driver=messagingv2
/var/lib/config-data/nova/etc/nova/nova.conf:driver=messagingv2
/var/lib/config-data/nova_metadata/etc/nova/nova.conf:driver=messagingv2
/var/lib/config-data/nova_placement/etc/nova/nova.conf:driver=messagingv2

()[rabbitmq@undercloud /]$ rabbitmqctl list_queues name consumers messages | grep notifications
notifications.warn	1	0
notifications.error	1	0
notifications.sample	1	0
notifications.debug	1	0
notifications.audit	1	0
notifications.critical	1	0
notifications.info	1	0