Description of problem: osp16.2 network creation fails (flaky) after controllers hard reboot Version-Release number of selected component (if applicable): OSP16.2 How reproducible: 100% , then try to create several networks Steps to Reproduce: 1. hard reboot all controllers 2. try to create several networks consecutively Actual results: some will succeed some will fail with: Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. Example : (overcloud) [stack@undercloud-0 ~]$ openstack network create internal_net_6832 Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create net_6832 Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_6832 +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2021-06-21T16:46:06Z | | description | | | dns_domain | | | id | 2830f099-c3da-42f2-977d-b70ca09f85a8 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='01b06020f25c43bb9bdaef89f51c6161', project.name='admin', region_name='regionOne', zone= | | mtu | 1442 | | name | bla_6832 | | port_security_enabled | True | | project_id | 01b06020f25c43bb9bdaef89f51c6161 | | provider:network_type | geneve | | provider:physical_network | None | | provider:segmentation_id | 5 | | qos_policy_id | None | | revision_number | 1 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2021-06-21T16:46:06Z | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack network create internal_bla_6832 Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_internal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_interna_bla +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2021-06-21T16:46:42Z | | description | | | dns_domain | | | id | e460b6ad-ebb4-4224-a228-b4728ba267e1 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='01b06020f25c43bb9bdaef89f51c6161', project.name='admin', region_name='regionOne', zone= | | mtu | 1442 | | name | bla_interna_bla | | port_security_enabled | True | | project_id | 01b06020f25c43bb9bdaef89f51c6161 | | provider:network_type | geneve | | provider:physical_network | None | | provider:segmentation_id | 6 | | qos_policy_id | None | | revision_number | 1 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2021-06-21T16:46:42Z | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_internal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_nternal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ternal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ernal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_rnal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_nal_bla +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2021-06-21T16:48:10Z | | description | | | dns_domain | | | id | 9b20fdc9-0ad2-4168-ac22-00493f4c0281 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='01b06020f25c43bb9bdaef89f51c6161', project.name='admin', region_name='regionOne', zone= | | mtu | 1442 | | name | bla_nal_bla | | port_security_enabled | True | | project_id | 01b06020f25c43bb9bdaef89f51c6161 | | provider:network_type | geneve | | provider:physical_network | None | | provider:segmentation_id | 7 | | qos_policy_id | None | | revision_number | 1 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2021-06-21T16:48:10Z | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_rnal_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ral_bla Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request. (overcloud) [stack@undercloud-0 ~]$ openstack network create bla_ral_bla Live env : seal15.qa.lab.tlv.redhat.com CI job used to deploy and test the env : https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/Phase3/view/OSP%2016.2/view/PidOne/job/DFG-pidone-sanity-16.2_director-rhel-virthost-3cont_2comp-ipv4-geneve-ansible-sts-sanity/
sosreports, stack home,all overcloud /var/log, are at : http://rhos-release.virt.bos.redhat.com/log/pkomarov_sosreports/BZ_1974439/
some errors I see a the time of he failures : [root@controller-0 neutron]# grep -i error server.log 2021-06-21 16:35:12.214 21 INFO neutron.pecan_wsgi.hooks.translation [req-ff27d961-4e16-4970-b584-437b3d600181 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] GET failed (client error): The resource could not be found. 2021-06-21 16:35:20.605 22 INFO neutron.pecan_wsgi.hooks.translation [req-03abe7ef-317e-443f-8a48-5771b9bff8e6 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] GET failed (client error): The resource could not be found. 2021-06-21 16:47:46.508 16 ERROR ovsdbapp.backend.ovs_idl.transaction [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] Traceback (most recent call last): raise KeyError(key) KeyError: UUID('1d6c6b48-f087-4cac-9813-e3b1608a38b5') 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] Mechanism driver 'ovn' failed in create_network_postcommit: KeyError: UUID('1d6c6b48-f087-4cac-9813-e3b1608a38b5') 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers yield self._nested_txns_map[cur_thread_id] 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers KeyError: 139699159308440 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred: 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/mech_driver.py", line 452, in create_network_postcommit 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers self._ovn_client.create_network(network) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1660, in create_network 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers self.create_provnet_port(network['id'], segment, txn=txn) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers next(self.gen) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 188, in transaction 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers yield t 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers next(self.gen) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers del self._nested_txns_map[cur_thread_id] 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers raise result.ex 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit()) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers self.post_commit(txn) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 70, in post_commit 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers command.post_commit(txn) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 90, in post_commit 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers row = self.api.tables[self.table_name].rows[real_uuid] 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python3.6/collections/__init__.py", line 991, in __getitem__ 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers raise KeyError(key) 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers KeyError: UUID('1d6c6b48-f087-4cac-9813-e3b1608a38b5') 2021-06-21 16:47:46.508 16 ERROR neutron.plugins.ml2.managers 2021-06-21 16:47:46.510 16 ERROR neutron.plugins.ml2.plugin [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] mechanism_manager.create_network_postcommit failed, deleting network '28ac8783-d966-4395-95f1-9454aa4df198': neutron.plugins.ml2.common.exceptions.MechanismDriverError 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation [req-e820a39c-da4b-498b-8a82-a06cd10d3a47 85d5fa283952484392440972f5cb2ed3 01b06020f25c43bb9bdaef89f51c6161 - default default] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 163, in post 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 181, in create 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)} 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 687, in inner 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1060, in create_network 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation return self._after_create_network(context, result, mech_context) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1071, in _after_create_network 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.delete_network(context, result['id']) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1066, in _after_create_network 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_network_postcommit(mech_context) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 527, in create_network_postcommit 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation self._call_on_drivers("create_network_postcommit", context) 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation errors=errors 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError 2021-06-21 16:47:46.872 16 ERROR neutron.pecan_wsgi.hooks.translation
reproducer : execute echo b> /proc/sysrq-trigger # on all controllers . overcloudrc execute the following a couple of times until it breaks: openstack network create net_some_name
probably dup or in relation to : https://bugzilla.redhat.com/show_bug.cgi?id=1965954 Since the fix of ^^ tested to resolve this issue also *** This bug has been marked as a duplicate of bug 1965954 ***
Verified , by creating multiple networks sequentially + full vm boot procedure, before and after all controllers reboots. This is a dup of: https://bugzilla.redhat.com/show_bug.cgi?id=1965954 The above causes multiple problems with neutron The fix of ^^ tested to resolve this issue also *** This bug has been marked as a duplicate of bug 1965954 ***
I am sorry, but maybe I misunderstood something. The patch mentioned in the other bug is not for ovsdbapp that means we are still having those issues and this does not solve anything for RHV. IIUC we should open new bug ourselves then?
Ales , yes I think so , to be on he safe side. I don't want openstack bugs to shadow any rhev bugs if they have fundemental diffences in how they are solved.