| Summary: | [OVN-OSP14] Cannot boot instance on OVN HA deployment | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eran Kuris <ekuris> | ||||
| Component: | openvswitch | Assignee: | Open vSwitch development team <ovs-team> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Eran Kuris <ekuris> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 14.0 (Rocky) | CC: | aconole, apevec, bcafarel, chrisw, fleitner, lhh, lmartins, majopela, nyechiel, ovs-team, rhos-maint, tvignaud, twilson | ||||
| Target Milestone: | beta | Keywords: | Triaged | ||||
| Target Release: | 14.0 (Rocky) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openvswitch2.10-2.10.0-18.el7fdn | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1639963 (view as bug list) | Environment: | |||||
| Last Closed: | 2019-01-11 11:53:11 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Bug Depends On: | 1639963 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
added a script as steps to reproduce Created attachment 1484097 [details]
create setup
I'm looking into this. It might be related to this fix for a race condition w/ greenthreads https://review.openstack.org/#/c/604029/ since it is raising an exception at a weird point. Exception happens in python-ovs, in __getattr__ which will show the "transaction view" of a Row. So if a row is modified in a transaction by removing a field from a dict, it won't show up when you access Row.column[field]. It looks like a field is moved from a dict in the transaction (it's in "removes") but the field is already no longer in the data. Still investigating. Fix verified : OpenStack/14.0-RHEL-7/2018-10-30.6 openvswitch2.10-2.10.0-21.el7fdn.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045 |
Description of problem: After a fresh install of OSP14vHA OVN deployment, I could not boot instances. The only way to solve the issue was to restart neutron API docker. 2018-09-17 15:21:27.001 1895 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCom│ mand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1- 5.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:21:27.004 1895 ERROR ovsdbapp.backend.ovs_idl.transaction [req-450772db-72fc-418c-8309-04fdf9240bc2 b2035 ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit command.run_idl(txn) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl if isinstance(getattr(record, self.column), dict): File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__ del dmap[key] KeyError: u'host-2001--f816-3eff-feff-b10c.ovn.test' kend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-2001--f816-3eff-fe7e-19c0',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-18.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2018-09-17 15:40:24.653 27 ERROR ovsdbapp.backend.ovs_idl.transaction [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit command.run_idl(txn) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl if isinstance(getattr(record, self.column), dict): File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__ del dmap[key] KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Mechanism driver 'ovn' failed in update_port_postcommit: KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 429, in _call_on_drivers 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 540, in update_port_postcommit 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self._ovn_client.update_port(port, port_object=original_port) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 489, in update_port 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.add_txns_to_remove_port_dns_records(txn, port_object) 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.gen.next() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 171, in transaction 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers yield t 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.gen.next() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 104, in transaction 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self._nested_txn = None 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 64, in __exit__ 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers raise result.ex 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers KeyError: u'host-2001--f816-3eff-fe7e-19c0' 2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers 2018-09-17 15:40:24.657 27 ERROR neutron.plugins.ml2.plugin [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] mechanism_manager.update_port_postcommit failed for port cbf60feb-b91c-487a-9c14-13dc6f3bbb00: MechanismDriverError Version-Release number of selected component (if applicable): OpenStack/14.0-RHEL-7/2018-09-06.1 ()[root@controller-0 /]# rpm -qa | grep -i ovn rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch openvswitch2.10-ovn-common-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 puppet-ovn-13.3.1-0.20180831170127.7f51c55.el7ost.noarch openvswitch2.10-ovn-central-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: