Bug 1964893

Summary: "NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception" error while instantiating instance in Train RDO
Product: [Community] RDO Reporter: startlearningnew
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED NOTABUG QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: trunkCC: dasmith, eglynn, jhakimra, kchamart, lyarwood, sbauza, sgordon, vromanso
Target Milestone: GA   
Target Release: trunk   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-26 10:07:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description startlearningnew 2021-05-26 09:44:38 UTC 
Description of problem:
Following error occurring while instantiating instance or healing operation on the Train RDO (the issue is random). No issue observed with Queen's RDO version.



2021-05-25 18:04:01.158 4217 ERROR nova.network.neutronv2.api [req-956c0f5c-3d06-4f70-ac5c-9b3c5b5f08b0 b88bbfabf5814caa905897192f5e48b8 518c8afd2aa448c68802c9e2957cfe8a - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi [req-956c0f5c-3d06-4f70-ac5c-9b3c5b5f08b0 b88bbfabf5814caa905897192f5e48b8 518c8afd2aa448c68802c9e2957cfe8a - default default] Unexpected exception in API method: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/attach_interfaces.py", line 76, in index
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     data = self.network_api.list_ports(context, **search_opts)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1699, in list_ports
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     return get_client(context).list_ports(**search_opts)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     **_params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     for r in self._pagination(collection, path, **params):
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     res = self.get(path, params=params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     headers=headers, params=params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     headers=headers, params=params)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     self._handle_fault_response(status_code, replybody, resp)
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi     raise exception.NeutronAdminCredentialConfigurationInvalid()
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-05-25 18:04:01.160 4217 ERROR nova.api.openstack.wsgi
2021-05-25 18:04:45.749 4261 WARNING keystonemiddleware.auth_token [req-f7eec590-5ed1-4dae-b2a5-a0f73fbbe921 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed


Also following errors thrown all the time, I am not sure anything related.

2021-05-25 18:44:49.432 4279 WARNING keystonemiddleware.auth_token [req-2d9ca2ca-bb59-4c70-99c0-1d341c24e439 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed
2021-05-25 18:44:49.991 4279 WARNING keystonemiddleware.auth_token [req-2d9ca2ca-bb59-4c70-99c0-1d341c24e439 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed
2021-05-25 18:49:44.213 4216 WARNING keystonemiddleware.auth_token [req-b915ed65-c6b3-4cfc-939f-6fb138ec9933 62ad0cd14e034f00892fb5adb0ffa293 6555e09485ef4a119b42e69d0b54e4e7 - default default] Authorization failed for token: InvalidToken: Token authorization failed
2021-05-25 18:49:47.388 4279 WARNING keystone



RabbitMQ log:
--------------

=ERROR REPORT==== 26-May-2021::09:07:44 ===
closing AMQP connection <0.13249.13> (10.232.14.48:38798 -> 10.232.14.48:5672 - mod_wsgi:4269:d80527ad-277b-4832-b60c-bba1d1c325b2):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 26-May-2021::09:07:44 ===
closing AMQP connection <0.13246.13> (10.232.14.48:38796 -> 10.232.14.48:5672 - mod_wsgi:4269:d5ff9555-e8f2-406b-a5c0-47e935cf8591):
missed heartbeats from client, timeout: 60s



Version-Release number of selected component (if applicable):
openstack-nova-api-20.6.0-1.el7.noarch
openstack-nova-common-20.6.0-1.el7.noarch
rabbitmq-server-3.6.16-1.el7.noarch
openstack-keystone-16.0.1-1.el7.noarch
openstack-dashboard-16.2.1-1.el7.noarch
openstack 4.0.2
Train RDO


How reproducible:
Instantiate or Heal the instance

Steps to Reproduce:
1. Instantiate the instance
2. Get the result of the instance after successfully created in Openstack

Actual results:
Instance created successfully, but above issue reported while fetching result


Expected results:
No error expected, the result should success

Additional info:
The issue didn't reported while using Queen's RDO
Comment 1 Lee Yarwood 2021-05-26 10:07:40 UTC 
As suggested by the error you need to ensure the credentials listed under the [neutron] section of the nova.conf used by the nova-api service are able to generate an admin token:

https://docs.openstack.org/nova/train/configuration/config.html#neutron

Moving this to the RDO product and closing it out as NOTABUG for the time being, if you are able to generate an admin token with the provided Neutron credentials then I'd recommend that you recreate this bug upstream https://launchpad.net/nova/+filebug as Red Hat provides no support for RDO downstream within this bug tracker.