Description of problem:
Following error occurring while instantiating instance or healing operation on the Train RDO (the issue is random). No issue observed with Queen's RDO version.



2021-05-25 18:04:01.158 4217 ERROR nova.network.neutronv2.api [req-956c0f5c-3d06-4f70-ac5c-9b3c5b5f08b0 b88bbfabf5814caa905897192f5e48b8 518c8afd2aa448c68802c9e2957cfe8a - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi [req-956c0f5c-3d06-4f70-ac5c-9b3c5b5f08b0 b88bbfabf5814caa905897192f5e48b8 518c8afd2aa448c68802c9e2957cfe8a - default default] Unexpected exception in API method: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/attach_interfaces.py", line 76, in index
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     data = self.network_api.list_ports(context, **search_opts)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1699, in list_ports
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     return get_client(context).list_ports(**search_opts)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     **_params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     for r in self._pagination(collection, path, **params):
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     res = self.get(path, params=params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     headers=headers, params=params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     headers=headers, params=params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     self._handle_fault_response(status_code, replybody, resp)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     raise exception.NeutronAdminCredentialConfigurationInvalid()
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi
2021-05-25 18:04:45.749 4261 WARNING keystonemiddleware.auth_token [req-f7eec590-5ed1-4dae-b2a5-a0f73fbbe921 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed


Also following errors thrown all the time, I am not sure anything related.

2021-05-25 18:44:49.432 4279 WARNING keystonemiddleware.auth_token [req-2d9ca2ca-bb59-4c70-99c0-1d341c24e439 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed
2021-05-25 18:44:49.991 4279 WARNING keystonemiddleware.auth_token [req-2d9ca2ca-bb59-4c70-99c0-1d341c24e439 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed
2021-05-25 18:49:44.213 4216 WARNING keystonemiddleware.auth_token [req-b915ed65-c6b3-4cfc-939f-6fb138ec9933 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed
2021-05-25 18:49:47.388 4279 WARNING keystone



RabbitMQ log:
--------------

=ERROR REPORT==== 26-May-2021::09:07:44 ===
closing AMQP connection <0.13249.13> (10.232.14.48:38798 -> 10.232.14.48:5672 - mod_wsgi:4269:d80527ad-277b-4832-b60c-bba1d1c325b2):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 26-May-2021::09:07:44 ===
closing AMQP connection <0.13246.13> (10.232.14.48:38796 -> 10.232.14.48:5672 - mod_wsgi:4269:d5ff9555-e8f2-406b-a5c0-47e935cf8591):
missed heartbeats from client, timeout: 60s



Version-Release number of selected component (if applicable):
openstack-nova-api-20.6.0-1.el7.noarch
openstack-nova-common-20.6.0-1.el7.noarch
rabbitmq-server-3.6.16-1.el7.noarch
openstack-keystone-16.0.1-1.el7.noarch
openstack-dashboard-16.2.1-1.el7.noarch
openstack 4.0.2
Train RDO


How reproducible:
Instantiate or Heal the instance

Steps to Reproduce:
1. Instantiate the instance
2. Get the result of the instance after successfully created in Openstack

Actual results:
Instance created successfully, but above issue reported while fetching result


Expected results:
No error expected, the result should success

Additional info:
The issue didn't reported while using Queen's RDO