Bug 1974439

Summary: osp16.2 network creation fails (flaky) after controllers hard reboot
Product: Red Hat OpenStack Reporter: pkomarov
Component: openstack-neutronAssignee: Terry Wilson <twilson>
Status: CLOSED DUPLICATE QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: amusil, atragler, bcafarel, chrisw, dciabrin, ksambor, lmartins, mburman, michele, mperina, scohen, spower
Target Milestone: betaKeywords: AutomationBlocker, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-07 12:18:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1940824    

Description pkomarov 2021-06-21 17:07:57 UTC 
Description of problem:
osp16.2 network creation fails (flaky) after controllers hard reboot

Version-Release number of selected component (if applicable):
OSP16.2

How reproducible:
100%

 , then try to create several networks

Steps to Reproduce:
1. hard reboot all controllers
2. try to create several networks consecutively


Actual results:
some will succeed some will fail with:

Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.

Example : 
(overcloud) [stack@undercloud-0 ~]$ openstack network create internal_net_6832
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create net_6832
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_6832
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                            |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                               |
| availability_zone_hints   |                                                                                                                                                                  |
| availability_zones        |                                                                                                                                                                  |
| created_at                | 2021-06-21T16:46:06Z                                                                                                                                             |
| description               |                                                                                                                                                                  |
| dns_domain                |                                                                                                                                                                  |
| id                        | 2830f099-c3da-42f2-977d-b70ca09f85a8                                                                                                                             |
| ipv4_address_scope        | None                                                                                                                                                             |
| ipv6_address_scope        | None                                                                                                                                                             |
| is_default                | False                                                                                                                                                            |
| is_vlan_transparent       | None                                                                                                                                                             |
| location                  | cloud='', project.domain_id=, project.domain_name='Default', project.id='01b06020f25c43bb9bdaef89f51c6161', project.name='admin', region_name='regionOne', zone= |
| mtu                       | 1442                                                                                                                                                             |
| name                      | bla_6832                                                                                                                                                         |
| port_security_enabled     | True                                                                                                                                                             |
| project_id                | 01b06020f25c43bb9bdaef89f51c6161                                                                                                                                 |
| provider:network_type     | geneve                                                                                                                                                           |
| provider:physical_network | None                                                                                                                                                             |
| provider:segmentation_id  | 5                                                                                                                                                                |
| qos_policy_id             | None                                                                                                                                                             |
| revision_number           | 1                                                                                                                                                                |
| router:external           | Internal                                                                                                                                                         |
| segments                  | None                                                                                                                                                             |
| shared                    | False                                                                                                                                                            |
| status                    | ACTIVE                                                                                                                                                           |
| subnets                   |                                                                                                                                                                  |
| tags                      |                                                                                                                                                                  |
| updated_at                | 2021-06-21T16:46:06Z                                                                                                                                             |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack network create internal_bla_6832
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_internal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_interna_bla
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                            |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                               |
| availability_zone_hints   |                                                                                                                                                                  |
| availability_zones        |                                                                                                                                                                  |
| created_at                | 2021-06-21T16:46:42Z                                                                                                                                             |
| description               |                                                                                                                                                                  |
| dns_domain                |                                                                                                                                                                  |
| id                        | e460b6ad-ebb4-4224-a228-b4728ba267e1                                                                                                                             |
| ipv4_address_scope        | None                                                                                                                                                             |
| ipv6_address_scope        | None                                                                                                                                                             |
| is_default                | False                                                                                                                                                            |
| is_vlan_transparent       | None                                                                                                                                                             |
| location                  | cloud='', project.domain_id=, project.domain_name='Default', project.id='01b06020f25c43bb9bdaef89f51c6161', project.name='admin', region_name='regionOne', zone= |
| mtu                       | 1442                                                                                                                                                             |
| name                      | bla_interna_bla                                                                                                                                                  |
| port_security_enabled     | True                                                                                                                                                             |
| project_id                | 01b06020f25c43bb9bdaef89f51c6161                                                                                                                                 |
| provider:network_type     | geneve                                                                                                                                                           |
| provider:physical_network | None                                                                                                                                                             |
| provider:segmentation_id  | 6                                                                                                                                                                |
| qos_policy_id             | None                                                                                                                                                             |
| revision_number           | 1                                                                                                                                                                |
| router:external           | Internal                                                                                                                                                         |
| segments                  | None                                                                                                                                                             |
| shared                    | False                                                                                                                                                            |
| status                    | ACTIVE                                                                                                                                                           |
| subnets                   |                                                                                                                                                                  |
| tags                      |                                                                                                                                                                  |
| updated_at                | 2021-06-21T16:46:42Z                                                                                                                                             |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_internal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_nternal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ternal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ernal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_rnal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_nal_bla
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                            |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                               |
| availability_zone_hints   |                                                                                                                                                                  |
| availability_zones        |                                                                                                                                                                  |
| created_at                | 2021-06-21T16:48:10Z                                                                                                                                             |
| description               |                                                                                                                                                                  |
| dns_domain                |                                                                                                                                                                  |
| id                        | 9b20fdc9-0ad2-4168-ac22-00493f4c0281                                                                                                                             |
| ipv4_address_scope        | None                                                                                                                                                             |
| ipv6_address_scope        | None                                                                                                                                                             |
| is_default                | False                                                                                                                                                            |
| is_vlan_transparent       | None                                                                                                                                                             |
| location                  | cloud='', project.domain_id=, project.domain_name='Default', project.id='01b06020f25c43bb9bdaef89f51c6161', project.name='admin', region_name='regionOne', zone= |
| mtu                       | 1442                                                                                                                                                             |
| name                      | bla_nal_bla                                                                                                                                                      |
| port_security_enabled     | True                                                                                                                                                             |
| project_id                | 01b06020f25c43bb9bdaef89f51c6161                                                                                                                                 |
| provider:network_type     | geneve                                                                                                                                                           |
| provider:physical_network | None                                                                                                                                                             |
| provider:segmentation_id  | 7                                                                                                                                                                |
| qos_policy_id             | None                                                                                                                                                             |
| revision_number           | 1                                                                                                                                                                |
| router:external           | Internal                                                                                                                                                         |
| segments                  | None                                                                                                                                                             |
| shared                    | False                                                                                                                                                            |
| status                    | ACTIVE                                                                                                                                                           |
| subnets                   |                                                                                                                                                                  |
| tags                      |                                                                                                                                                                  |
| updated_at                | 2021-06-21T16:48:10Z                                                                                                                                             |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_rnal_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ral_bla
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.
(overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ral_bla

Live env : 
seal15.qa.lab.tlv.redhat.com

CI job used to deploy and test the env : 

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/Phase3/view/OSP%2016.2/view/PidOne/job/DFG-pidone-sanity-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve-ansible-sts-sanity/
Comment 1 pkomarov 2021-06-21 17:43:30 UTC 
sosreports, stack home,all overcloud /var/log, are at : http://rhos-release.virt.bos.redhat.com/log/pkomarov_sosreports/BZ_1974439/
Comment 3 pkomarov 2021-06-21 18:12:30 UTC 
some errors I see a the time of he failures : 

[root@controller-0 neutron]# grep -i error server.log
2021-06-21 16:35:12.214 21 INFO neutron.pecan_wsgi.hooks.translation [req-ff27d961-4e16-4970-b584-437b3d600181 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] GET failed (client error): The resource could not be found.
2021-06-21 16:35:20.605 22 INFO neutron.pecan_wsgi.hooks.translation [req-03abe7ef-317e-443f-8a48-5771b9bff8e6 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] GET failed (client error): The resource could not be found.
2021-06-21 16:47:46.508 16 ERROR ovsdbapp.backend.ovs_idl.transaction [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] Traceback (most recent call last):
    raise KeyError(key)
KeyError: UUID('1d6c6b48-f087-4cac-9813-e3b1608a38b5')
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] Mechanism driver 'ovn' failed in create_network_postcommit: KeyError: UUID('1d6c6b48-f087-4cac-9813-e3b1608a38b5')
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     yield self._nested_txns_map[cur_thread_id]
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers KeyError: 139699159308440
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers 
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers 
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/mech_driver.py", line 452, in create_network_postcommit
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     self._ovn_client.create_network(network)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1660, in create_network
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     self.create_provnet_port(network['id'], segment, txn=txn)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     next(self.gen)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 188, in transaction
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     yield t
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     next(self.gen)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     del self._nested_txns_map[cur_thread_id]
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     raise result.ex
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     txn.results.put(txn.do_commit())
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     self.post_commit(txn)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 70, in post_commit
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     command.post_commit(txn)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 90, in post_commit
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     row = self.api.tables[self.table_name].rows[real_uuid]
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.6/collections/__init__.py", line 991, in __getitem__
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers     raise KeyError(key)
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers KeyError: UUID('1d6c6b48-f087-4cac-9813-e3b1608a38b5')
2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers 
2021-06-21 16:47:46.510 16 ERROR neutron.plugins.ml2.plugin [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] mechanism_manager.create_network_postcommit failed, deleting network '28ac8783-d966-4395-95f1-9454aa4df198': neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return self.create(resources)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return {key: creator(*creator_args, **creator_kwargs)}
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 687, in inner
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1060, in create_network
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     return self._after_create_network(context, result, mech_context)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1071, in _after_create_network
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.delete_network(context, result['id'])
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     raise value
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1066, in _after_create_network
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self.mechanism_manager.create_network_postcommit(mech_context)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 527, in create_network_postcommit
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     self._call_on_drivers("create_network_postcommit", context)
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation     errors=errors
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation
Comment 4 pkomarov 2021-06-22 07:58:48 UTC 
reproducer :

execute echo b> /proc/sysrq-trigger # on all controllers

. overcloudrc

execute the following a couple of times until it breaks:

openstack network create net_some_name
Comment 8 pkomarov 2021-07-07 12:18:20 UTC 
probably dup or in relation to : 
https://bugzilla.redhat.com/show_bug.cgi?id=1965954
Since the fix of ^^ tested to resolve this issue also

*** This bug has been marked as a duplicate of bug 1965954 ***
Comment 10 pkomarov 2021-07-08 06:28:23 UTC 
Verified , by creating multiple networks sequentially + full vm boot procedure, before and after all controllers reboots.
This is a dup of: 
https://bugzilla.redhat.com/show_bug.cgi?id=1965954
The above causes multiple problems with neutron
The fix of ^^ tested to resolve this issue also

*** This bug has been marked as a duplicate of bug 1965954 ***
Comment 11 Ales Musil 2021-07-08 06:33:20 UTC 
I am sorry, but maybe I misunderstood something.
The patch mentioned in the other bug is not for ovsdbapp that means 
we are still having those issues and this does not solve anything for RHV.
IIUC we should open new bug ourselves then?
Comment 12 pkomarov 2021-07-08 06:45:50 UTC 
Ales , yes I think so , to be on he safe side. 
I don't want openstack bugs to shadow any rhev bugs if they have fundemental diffences in how they are solved.