2124874 – Neutron can't connect to the OVN after tobiko faults tests

Bug 2124874 - Neutron can't connect to the OVN after tobiko faults tests

Summary: Neutron can't connect to the OVN after tobiko faults tests

Keywords:
Status:	CLOSED DUPLICATE of bug 2124877
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	16.2 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Rodolfo Alonso
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-09-07 10:45 UTC by Slawek Kaplonski
Modified:	2022-09-28 07:34 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-09-28 07:34:49 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-18588	0	None	None	None	2022-09-07 11:03:38 UTC

Description Slawek Kaplonski 2022-09-07 10:45:47 UTC

Failed Tobiko job results: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/gate-sanity-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve-tobiko_faults/521/test_results/tobiko_gate_1/tobiko_gate_1_06_after_faults_sanity.html

After tobiko faults phase is run, where there is many distruptions, like restarting nodes done, in next phase tobiko tries to create network and it fails with internal server error from Neutron.
In the neutron logs (http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/gate-sanity-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve-tobiko_faults/521/controller-1/var/log/containers/neutron/server.log.1.gz) You can find errors like:

2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager Traceback (most recent call last):
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 55, in commit
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     result = self.results.get(timeout=self.timeout)
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     return waiter.wait()
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     return get_hub().switch()
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     return self.greenlet.switch()
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager queue.Empty
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager 
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager During handling of the above exception, another exception occurred:
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager 
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager Traceback (most recent call last):
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/neutron/services/logapi/drivers/manager.py", line 116, in call
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     getattr(driver, method_name)(*args, **kwargs)
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/logapi_driver.py", line 285, in create_log
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     utils.ovn_name(log_obj.id))
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     next(self.gen)
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 252, in transaction
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     yield t
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     next(self.gen)
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     del self._nested_txns_map[cur_thread_id]
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     self.result = self.commit()
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 58, in commit
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager     timeout=self.timeout)
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager ovsdbapp.exceptions.TimeoutException: Commands [<ovsdbapp.schema.ovn_northbound.commands.MeterAddCommand object at 0x7fb0c3f69c18>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7fb0c3f69080>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7fb0c3f699b0>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7fb0c3f69f60>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7fb0c3f692b0>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7fb0c3f69908>] exceeded timeout 180 seconds
2022-09-06 03:02:01.350 19 ERROR neutron.services.logapi.drivers.manager 
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource [req-6718c9d8-c062-4c65-aa6d-262645d62a79 98a7c8cfd6c24b57a2daf66f02b50aaf 2e8916a3af054db8944550c6c4e3898a - default default] create failed: No details.: neutron_lib.exceptions.DriverCallError
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource Traceback (most recent call last):
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 437, in create
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     self.force_reraise()
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     raise value
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     self.force_reraise()
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     raise value
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     self.force_reraise()
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     raise value
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 556, in _create
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     obj = do_create(body)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 538, in do_create
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     self.force_reraise()
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     raise value
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 531, in do_create
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/db_base_plugin_common.py", line 49, in inner
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     result = f(*args, **kwargs)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/services/logapi/logging_plugin.py", line 112, in create_log
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     log_const.CREATE_LOG, context, log_obj)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/services/logapi/drivers/manager.py", line 125, in call
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource     raise exceptions.DriverCallError(exc_list=exc_list)
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource neutron_lib.exceptions.DriverCallError
2022-09-06 03:02:01.354 19 ERROR neutron.api.v2.resource 
2022-09-06 03:02:01.358 19 INFO neutron.wsgi [req-6718c9d8-c062-4c65-aa6d-262645d62a79 98a7c8cfd6c24b57a2daf66f02b50aaf 2e8916a3af054db8944550c6c4e3898a - default default] 10.0.0.16,172.17.1.133 "POST /v2.0/log/logs HTTP/1.1" status: 500  len: 344 time: 180.0495083

Comment 1 Slawek Kaplonski 2022-09-28 07:34:49 UTC


*** This bug has been marked as a duplicate of bug 2124877 ***

Note You need to log in before you can comment on or make changes to this bug.