Description of problem:

Getting "ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected" in /var/log/containers/neutron/server.log after enabling CADF for neutron specifically.



Version-Release number of selected component (if applicable):

[heat-admin@overcloud-controller-0 ~]$ rpm -qa | grep neutron
puppet-neutron-12.4.1-4.ed05e01git.el7ost.noarch
openstack-neutron-linuxbridge-12.0.5-4.el7ost.noarch
python2-neutronclient-6.7.0-1.el7ost.noarch
openstack-neutron-common-12.0.5-4.el7ost.noarch
python-neutron-lbaas-12.0.1-0.20181019202914.b9b6b6a.el7ost.noarch
openstack-neutron-lbaas-12.0.1-0.20181019202914.b9b6b6a.el7ost.noarch
openstack-neutron-openvswitch-12.0.5-4.el7ost.noarch
openstack-neutron-metering-agent-12.0.5-4.el7ost.noarch
openstack-neutron-lbaas-ui-4.0.1-0.20181115043347.7f2010d.el7ost.noarch
python-neutron-12.0.5-4.el7ost.noarch
openstack-neutron-ml2-12.0.5-4.el7ost.noarch
openstack-neutron-12.0.5-4.el7ost.noarch
openstack-neutron-l2gw-agent-12.0.2-0.20180412115803.a9f8009.el7ost.noarch
python2-neutron-lib-1.13.0-1.el7ost.noarch
openstack-neutron-sriov-nic-agent-12.0.5-4.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:

Steps to reproduce = Env (1 compute + 1 controller)

If multiple controller nodes the the changes has to be done on all the controller nodes.


1. On Controller configure api-paste.ini which supports CADF audits:

1.1 Copy the api-paste.ini from the distribution into the container to /etc/neutron (as we need to add some options):
[root@overcloud-controller-0 ~]# cp /usr/share/neutron/api-paste.ini /var/lib/config-data/puppet-generated/neutron/etc/neutron/api-paste.ini

1.2 Add the "audit" filter to the pipeline (this only adds the work "audit" into the exisiting parameter):
[root@overcloud-controller-0 ~]# crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/api-paste.ini composite:neutronapi_v2_0 keystone "cors http_proxy_to_wsgi request_id catch_errors authtoken audit keystonecontext extensions neutronapiapp_v2_0"

1.3 Add the filter conguration for the audit filter:
[root@overcloud-controller-0 ~]# crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/api-paste.ini  filter:audit paste.filter_factory keystonemiddleware.audit:filter_factory
[root@overcloud-controller-0 ~]# crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/api-paste.ini filter:audit audit_map_file /etc/pycadf/neutron_api_audit_map.conf

1.4 Configure neutron.conf to use this updated api-paste.ini:
[root@overcloud-controller-0 ~]# crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT api_paste_config /etc/neutron/api-paste.ini

2. Verify notifications are enabled (this should be the case unless it's been explicitly disabled):
[root@overcloud-controller-0 ~]# grep oslo_messaging_notifications -A17 /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf 
[oslo_messaging_notifications]

#
# From oslo.messaging
#

# The Drivers(s) to handle sending notifications. Possible values are
# messaging, messagingv2, routing, log, test, noop (multi valued)
# Deprecated group/name - [DEFAULT]/notification_driver
#driver =
driver=messagingv2

# A URL representing the messaging driver to use for notifications. If not set,
# we fall back to the same configuration used for RPC. (string value)
# Deprecated group/name - [DEFAULT]/notification_transport_url
#transport_url = <None>
transport_url=rabbit://guest:PASSWORD.localdomain:5672/?ssl=0

3. Restart the neutron_api container:
[root@overcloud-controller-0 ~]# docker restart neutron_api

4. Create a network:
(overcloud) [stack@undercloud ~]$ openstack network create audittest

5. Check the logs on the controller:
[root@overcloud-controller-0 ~]# less /var/log/containers/neutron/server.log
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging [req-13fcf074-7b38-44ce-8cd1-23379a820140 8705c7205e87461eba13a19963318574 027cffa5b5db4fd8a84df958d4c2afe5 - default default] Could not send notification to notifications. Payload={'event_type': 'audit.http.response', 'timestamp': u'2019-04-24 12:01:35.450291', '_context_query_params': {'page_reverse': False, 'limit': None, 'filters': {u'tenant_id': [u'027cffa5b5db4fd8a84df958d4c2afe5'], u'device_id': [u'84c7c0be-858d-4ae4-944a-06c6faaf633a']}, 'marker': None, 'fields': [], 'sorts': [('id', True)]}, '_unique_id': 'e48bb4edb8414c0fa14c098bdf835266', '_context_collection': 'ports', '_context_neutron_context': <neutron_lib.context.Context object at 0x7f68f3e1a190>, '_context_resource': 'port', '_context_pagination_helper': <neutron.api.api_common.PaginationNativeHelper object at 0x7f68f38d6310>, 'payload': {'typeURI': 'http://schemas.dmtf.org/cloud/audit/1.0/event', 'eventTime': '2019-04-24T12:01:35.038250+0000', 'target': {'typeURI': 'service/network/ports', 'addresses': [{'url': u'http://172.17.1.150:9696', 'name': 'admin'}, {'url': u'http://172.17.1.150:9696', 'name': 'private'}, {'url': u'http://192.168.122.150:9696', 'name': 'public'}], 'name': u'neutron', 'id': u'neutron'}, 'observer': {'id': 'target'}, 'tags': ['correlation_id?value=98a32aca-c274-5c99-bab9-937f2fe998b2'], 'eventType': 'activity', 'initiator': {'typeURI': 'service/security/account/user', 'name': u'neutron', 'credential': {'token': 'gAAAAABcwEXXERoW0brlD5T xxxxxxxx r588Yc-RGn-XqSmYmxbARS8', 'identity_status': 'Confirmed'}, 'host': {'agent': 'python-neutronclient', 'address': '172.17.1.201'}, 'project_id': u'027cffa5b5db4fd8a84df958d4c2afe5', 'id': u'8705c7205e87461eba13a19963318574'}, 'reason': {'reasonCode': '200', 'reasonType': 'HTTP'}, 'reporterchain': [{'reporterTime': '2019-04-24T12:01:35.448866+0000', 'role': 'modifier', 'reporter': {'id': 'target'}}], 'action': 'read/list', 'outcome': 'success', 'id': '1ff09d51-b9f8-5c11-b4f5-6b3353c9ac98', 'requestPath': '/v2.0/ports?tenant_id=027cffa5b5db4fd8a84df958d4c2afe5&device_id=84c7c0be-858d-4ae4-944a-06c6faaf633a'}, '_context_added_fields': [], '_context_sorting_helper': <neutron.api.api_common.SortingNativeHelper object at 0x7f68f38d66d0>, 'priority': 'INFO', '_context_uri_identifiers': {}, 'publisher_id': 'neutron-server', 'message_id': u'becc8388-f734-42f8-a27d-0ead7cbe41f9'}: ValueError: Circular reference detected
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/notify/messaging.py", line 70, in notify
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     retry=retry)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 138, in _send_notification
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     retry=retry)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 563, in send_notification
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     envelope=(version == 2.0), notify=True, retry=retry)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 511, in _send
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     msg = rpc_common.serialize_msg(msg)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 293, in serialize_msg
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 213, in dumps
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     return json.dumps(obj, default=default, **kwargs)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     sort_keys=sort_keys, **kw).encode(obj)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     chunks = self.iterencode(o, _one_shot=True)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging     return _iterencode(o, 0)
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected
2019-04-24 12:01:35.451 85 ERROR oslo_messaging.notify.messaging 


Actual results:

Facing "ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected"

Expected results:

The error is not expected.


Additional info: