Bug 1629934 - [OVN-OSP14] Cannot boot instance on OVN HA deployment
Summary: [OVN-OSP14] Cannot boot instance on OVN HA deployment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openvswitch
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 14.0 (Rocky)
Assignee: Open vSwitch development team
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On: 1639963
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-17 16:10 UTC by Eran Kuris
Modified: 2019-10-23 13:07 UTC (History)
13 users (show)

Fixed In Version: openvswitch2.10-2.10.0-18.el7fdn
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1639963 (view as bug list)
Environment:
Last Closed: 2019-01-11 11:53:11 UTC
Target Upstream Version:


Attachments (Terms of Use)
create setup (6.34 KB, application/x-shellscript)
2018-09-17 16:17 UTC, Eran Kuris
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:0045 None None None 2019-01-11 11:53:20 UTC

Internal Links: 1719099

Description Eran Kuris 2018-09-17 16:10:35 UTC
Description of problem:
After a fresh install of OSP14vHA OVN deployment, I could not boot instances.
The only way to solve the issue was to restart neutron API docker.

2018-09-17 15:21:27.001 1895 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCom│
 mand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-
 5.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84        
 2018-09-17 15:21:27.004 1895 ERROR ovsdbapp.backend.ovs_idl.transaction [req-450772db-72fc-418c-8309-04fdf9240bc2 b2035
 ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last):     
   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run                      
     txn.results.put(txn.do_commit())                                                                                   
   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit              
     command.run_idl(txn)                                                                                               
   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl                  
     if isinstance(getattr(record, self.column), dict):                                                                
     File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__                  
     del dmap[key]                    KeyError: u'host-2001--f816-3eff-feff-b10c.ovn.test'  
kend/ovs_idl/transaction.py:84
  2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-2001--f816-3eff-fe7e-19c0',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-18.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2018-09-17 15:40:24.653 27 ERROR ovsdbapp.backend.ovs_idl.transaction [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
      command.run_idl(txn)
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl
      if isinstance(getattr(record, self.column), dict):
    File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__
      del dmap[key]
  KeyError: u'host-2001--f816-3eff-fe7e-19c0'
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Mechanism driver 'ovn' failed in update_port_postcommit: KeyError: u'host-2001--f816-3eff-fe7e-19c0'
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 429, in _call_on_drivers
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 540, in update_port_postcommit
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self._ovn_client.update_port(port, port_object=original_port)
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 489, in update_port
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.add_txns_to_remove_port_dns_records(txn, port_object)
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.gen.next()
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 171, in transaction
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     yield t
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.gen.next()
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 104, in transaction
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self._nested_txn = None
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 64, in __exit__
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     raise result.ex
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers KeyError: u'host-2001--f816-3eff-fe7e-19c0'
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers 
  2018-09-17 15:40:24.657 27 ERROR neutron.plugins.ml2.plugin [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] mechanism_manager.update_port_postcommit failed for port cbf60feb-b91c-487a-9c14-13dc6f3bbb00: MechanismDriverError
Version-Release number of selected component (if applicable):
OpenStack/14.0-RHEL-7/2018-09-06.1
()[root@controller-0 /]# rpm -qa | grep -i ovn 
rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch
rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch
openvswitch2.10-ovn-common-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
puppet-ovn-13.3.1-0.20180831170127.7f51c55.el7ost.noarch
openvswitch2.10-ovn-central-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Eran Kuris 2018-09-17 16:16:31 UTC
added a script as steps to reproduce

Comment 3 Eran Kuris 2018-09-17 16:17:01 UTC
Created attachment 1484097 [details]
create setup

Comment 4 Terry Wilson 2018-09-24 13:00:41 UTC
I'm looking into this. It might be related to this fix for a race condition w/ greenthreads https://review.openstack.org/#/c/604029/ since it is raising an exception at a weird point.

Exception happens in python-ovs, in __getattr__ which will show the "transaction view" of a Row. So if a row is modified in a transaction by removing a field from a dict, it won't show up when you access Row.column[field]. It looks like a field is moved from a dict in the transaction (it's in "removes") but the field is already no longer in the data. Still investigating.

Comment 14 Eran Kuris 2018-11-01 11:53:12 UTC
Fix verified : 
OpenStack/14.0-RHEL-7/2018-10-30.6
openvswitch2.10-2.10.0-21.el7fdn.x86_64

Comment 17 errata-xmlrpc 2019-01-11 11:53:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.