1947290 – router list and creations replied with 500 server error after overcloud nodes reboot

Bug 1947290 - router list and creations replied with 500 server error after overcloud nodes reboot

Summary: router list and creations replied with 500 server error after overcloud nodes...

Keywords:
Status:	CLOSED DUPLICATE of bug 2119098
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-networking-ovn
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	z9
Target Release:	16.1 (Train on RHEL 8.2)
Assignee:	Jakub Libosvar
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:	2068271
Blocks:
TreeView+	depends on / blocked

Reported:	2021-04-08 07:26 UTC by Eduardo Olivares
Modified:	2022-09-13 20:39 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2068271 (view as bug list)
Environment:
Last Closed:	2022-09-13 20:39:47 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	781555	0	None	MERGED	Add wait for the post-fork event to nb/sb objects	2022-05-06 21:27:10 UTC
Red Hat Issue Tracker	OSP-2002	0	None	None	None	2021-11-17 09:37:40 UTC

Description Eduardo Olivares 2021-04-08 07:26:43 UTC

Description of problem:
Issue reproduced with OSP16.1 update job [0].

This job performs the following stages:
1) installs OSP16.1
2) runs tempest -> success
3) updates to a later OSP16.1 build
4) runs tempest -> success
5) reboots overcloud nodes
6) runs tempest -> failed

All the previous tempest stages include the task "Save the current state for the later cleanup". That task is successful at stages 2) and 4), but fails at stage 6) when tempest tries to obtain a list of routers [1]:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 109, in take_action
    self.init(parsed_args)
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 146, in init
    self._init_state()
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 285, in _init_state
    svc.run()
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup_service.py", line 134, in run
    self.save_state()
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup_service.py", line 523, in save_state
    routers = self.list()
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup_service.py", line 484, in list
    routers = client.list_routers(**self.tenant_filter)
  File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/routers_client.py", line 62, in list_routers
    return self.list_resources(uri, **filters)
  File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 34, in list_resources
    resp, body = self.get(req_uri)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 298, in get
    return self.request('GET', url, extra_headers, headers)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 864, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.


The following error can be seen in neutron server logs [2]:
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource [req-563fc04e-344e-45fd-8602-671ef568ee33 c350cb0abc0d498eba104d9693d70552 12d387ca1ccd424889a6f1e4b78b7897 - default default] index failed: No details.: AttributeError: 'NoneType' object has no attribute 'get_lrouter'
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource Traceback (most recent call last):
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     self.force_reraise()
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     raise value
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     self.force_reraise()
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     raise value
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     self.force_reraise()
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     raise value
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 369, in index
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return self._items(request, True, parent_id)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 304, in _items
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     obj_list = obj_getter(request.context, **kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     self.force_reraise()
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     raise value
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     self.force_reraise()
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     raise value
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     self.force_reraise()
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     raise value
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 548, in get_routers
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     page_reverse=page_reverse)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 311, in get_collection
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     for c in query
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 311, in <listcomp>
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     for c in query
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 221, in _make_router_dict
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     resource_extend.apply_funcs(l3_apidef.ROUTERS, res, router)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/resource_extend.py", line 84, in apply_funcs
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     resolved_func(response, db_object)
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/availability_zone/router.py", line 40, in _add_az_to_response
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     l3_plugin.get_router_availability_zones(router_db))
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/networking_ovn/l3/l3_ovn.py", line 450, in get_router_availability_zones
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource     lr = self._ovn.get_lrouter(router['id'])
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource AttributeError: 'NoneType' object has no attribute 'get_lrouter'
2021-04-07 17:43:55.636 25 ERROR neutron.api.v2.resource 
2021-04-07 17:43:55.643 25 INFO neutron.wsgi [req-563fc04e-344e-45fd-8602-671ef568ee33 c350cb0abc0d498eba104d9693d70552 12d387ca1ccd424889a6f1e4b78b7897 - default default] 172.17.1.104 "GET /v2.0/routers HTTP/1.1" status: 500  len: 363 time: 0.3538365






[0]  https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-update-16.1_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/58/
[1] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-update-16.1_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/58/console_logs/ir-tempest-3.log
[2] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-update-16.1_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/58/controller-0/var/log/containers/neutron/server.log.gz


Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20210323.n.0


How reproducible:
Only tried once and reproduced once.

Steps to Reproduce:
1. rerun the job [0]
2.
3.

Comment 3 Jakub Libosvar 2022-01-26 22:15:46 UTC

Terry fixed it upstream: https://review.opendev.org/c/openstack/neutron/+/781555

Comment 8 Jakub Libosvar 2022-09-13 20:39:47 UTC


*** This bug has been marked as a duplicate of bug 2119098 ***

Note You need to log in before you can comment on or make changes to this bug.