I'm looking into this. It might be related to this fix for a race condition w/ greenthreads https://review.openstack.org/#/c/604029/ since it is raising an exception at a weird point.
Exception happens in python-ovs, in __getattr__ which will show the "transaction view" of a Row. So if a row is modified in a transaction by removing a field from a dict, it won't show up when you access Row.column[field]. It looks like a field is moved from a dict in the transaction (it's in "removes") but the field is already no longer in the data. Still investigating.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2019:0045
Description of problem: After a fresh install of OSP14vHA OVN deployment, I could not boot instances. The only way to solve the issue was to restart neutron API docker. 2018-09-17 15:21:27.001 1895 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCom│ mand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1- 5.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:21:27.004 1895 ERROR ovsdbapp.backend.ovs_idl.transaction [req-450772db-72fc-418c-8309-04fdf9240bc2 b2035 ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit command.run_idl(txn) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl if isinstance(getattr(record, self.column), dict): File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__ del dmap[key] KeyError: u'host-2001--f816-3eff-feff-b10c.ovn.test' kend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-2001--f816-3eff-fe7e-19c0',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-18.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.653 27 ERROR ovsdbapp.backend.ovs_idl.transaction [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit command.run_idl(txn) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl if isinstance(getattr(record, self.column), dict): File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__ del dmap[key] KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Mechanism driver 'ovn' failed in update_port_postcommit: KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 429, in _call_on_drivers 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 540, in update_port_postcommit 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self._ovn_client.update_port(port, port_object=original_port) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 489, in update_port 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.add_txns_to_remove_port_dns_records(txn, port_object) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.gen.next() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 171, in transaction 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers yield t 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.gen.next() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 104, in transaction 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self._nested_txn = None 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 64, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers raise result.ex 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers 2018-09-17 15:40:24.657 27 ERROR neutron.plugins.ml2.plugin [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] mechanism_manager.update_port_postcommit failed for port cbf60feb-b91c-487a-9c14-13dc6f3bbb00: MechanismDriverError Version-Release number of selected component (if applicable): OpenStack/14.0-RHEL-7/2018-09-06.1 ()[root@controller-0 /]# rpm -qa | grep -i ovn rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch openvswitch2.10-ovn-common-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 puppet-ovn-13.3.1-0.20180831170127.7f51c55.el7ost.noarch openvswitch2.10-ovn-central-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: