Hide Forgot
Description of problem: After a fresh install of OSP14vHA OVN deployment, I could not boot instances. The only way to solve the issue was to restart neutron API docker. 2018-09-17 15:21:27.001 1895 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCom│ mand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1- 5.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:21:27.004 1895 ERROR ovsdbapp.backend.ovs_idl.transaction [req-450772db-72fc-418c-8309-04fdf9240bc2 b2035 ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit command.run_idl(txn) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl if isinstance(getattr(record, self.column), dict): File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__ del dmap[key] KeyError: u'host-2001--f816-3eff-feff-b10c.ovn.test' kend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-2001--f816-3eff-fe7e-19c0',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-18.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.653 27 ERROR ovsdbapp.backend.ovs_idl.transaction [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit command.run_idl(txn) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl if isinstance(getattr(record, self.column), dict): File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__ del dmap[key] KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Mechanism driver 'ovn' failed in update_port_postcommit: KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 429, in _call_on_drivers 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 540, in update_port_postcommit 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self._ovn_client.update_port(port, port_object=original_port) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 489, in update_port 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.add_txns_to_remove_port_dns_records(txn, port_object) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.gen.next() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 171, in transaction 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers yield t 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.gen.next() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 104, in transaction 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self._nested_txn = None 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 64, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers raise result.ex 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers 2018-09-17 15:40:24.657 27 ERROR neutron.plugins.ml2.plugin [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] mechanism_manager.update_port_postcommit failed for port cbf60feb-b91c-487a-9c14-13dc6f3bbb00: MechanismDriverError Version-Release number of selected component (if applicable): OpenStack/14.0-RHEL-7/2018-09-06.1 ()[root@controller-0 /]# rpm -qa | grep -i ovn rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch openvswitch2.10-ovn-common-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 puppet-ovn-13.3.1-0.20180831170127.7f51c55.el7ost.noarch openvswitch2.10-ovn-central-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
added a script as steps to reproduce
Created attachment 1484097 [details] create setup
I'm looking into this. It might be related to this fix for a race condition w/ greenthreads https://review.openstack.org/#/c/604029/ since it is raising an exception at a weird point. Exception happens in python-ovs, in __getattr__ which will show the "transaction view" of a Row. So if a row is modified in a transaction by removing a field from a dict, it won't show up when you access Row.column[field]. It looks like a field is moved from a dict in the transaction (it's in "removes") but the field is already no longer in the data. Still investigating.
Fix verified : OpenStack/14.0-RHEL-7/2018-10-30.6 openvswitch2.10-2.10.0-21.el7fdn.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045