Bug 1629934

Summary: [OVN-OSP14] Cannot boot instance on OVN HA deployment
Product: Red Hat OpenStack Reporter: Eran Kuris <ekuris>
Component: openvswitchAssignee: Open vSwitch development team <ovs-team>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 14.0 (Rocky)CC: aconole, apevec, bcafarel, chrisw, fleitner, lhh, lmartins, majopela, nyechiel, ovs-team, rhos-maint, tvignaud, twilson
Target Milestone: betaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.10-2.10.0-18.el7fdn Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1639963 (view as bug list) Environment:
Last Closed: 2019-01-11 11:53:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1639963    
Bug Blocks:    
Attachments:
Description Flags
create setup none

Description Eran Kuris 2018-09-17 16:10:35 UTC
Description of problem:
After a fresh install of OSP14vHA OVN deployment, I could not boot instances.
The only way to solve the issue was to restart neutron API docker.

2018-09-17 15:21:27.001 1895 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCom│
 mand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-
 5.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84        
 2018-09-17 15:21:27.004 1895 ERROR ovsdbapp.backend.ovs_idl.transaction [req-450772db-72fc-418c-8309-04fdf9240bc2 b2035
 ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last):     
   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run                      
     txn.results.put(txn.do_commit())                                                                                   
   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit              
     command.run_idl(txn)                                                                                               
   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl                  
     if isinstance(getattr(record, self.column), dict):                                                                
     File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__                  
     del dmap[key]                    KeyError: u'host-2001--f816-3eff-feff-b10c.ovn.test'  
kend/ovs_idl/transaction.py:84
  2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-2001--f816-3eff-fe7e-19c0',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2018-09-17 15:40:24.652 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbRemoveCommand(if_exists=False, column=records, keyvalues={}, record=61f6f840-f641-4560-8367-83995dbf72bd, values=(u'host-10-0-1-18.ovn.test',), table=DNS) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2018-09-17 15:40:24.653 27 ERROR ovsdbapp.backend.ovs_idl.transaction [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
      command.run_idl(txn)
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl
      if isinstance(getattr(record, self.column), dict):
    File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 841, in __getattr__
      del dmap[key]
  KeyError: u'host-2001--f816-3eff-fe7e-19c0'
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] Mechanism driver 'ovn' failed in update_port_postcommit: KeyError: u'host-2001--f816-3eff-fe7e-19c0'
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 429, in _call_on_drivers
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 540, in update_port_postcommit
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self._ovn_client.update_port(port, port_object=original_port)
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 489, in update_port
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.add_txns_to_remove_port_dns_records(txn, port_object)
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.gen.next()
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 171, in transaction
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     yield t
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.gen.next()
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 104, in transaction
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self._nested_txn = None
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 64, in __exit__
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     self.result = self.commit()
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers     raise result.ex
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers KeyError: u'host-2001--f816-3eff-fe7e-19c0'
  2018-09-17 15:40:24.654 27 ERROR neutron.plugins.ml2.managers 
  2018-09-17 15:40:24.657 27 ERROR neutron.plugins.ml2.plugin [req-de466a87-dba2-461f-ad61-d70db82fc799 b2035ad60a08494097c09b38ab090031 e6cbe8175d1041dcbeac33cb6353b1ff - default default] mechanism_manager.update_port_postcommit failed for port cbf60feb-b91c-487a-9c14-13dc6f3bbb00: MechanismDriverError
Version-Release number of selected component (if applicable):
OpenStack/14.0-RHEL-7/2018-09-06.1
()[root@controller-0 /]# rpm -qa | grep -i ovn 
rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch
rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch
openvswitch2.10-ovn-common-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
puppet-ovn-13.3.1-0.20180831170127.7f51c55.el7ost.noarch
openvswitch2.10-ovn-central-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Eran Kuris 2018-09-17 16:16:31 UTC
added a script as steps to reproduce

Comment 3 Eran Kuris 2018-09-17 16:17:01 UTC
Created attachment 1484097 [details]
create setup

Comment 4 Terry Wilson 2018-09-24 13:00:41 UTC
I'm looking into this. It might be related to this fix for a race condition w/ greenthreads https://review.openstack.org/#/c/604029/ since it is raising an exception at a weird point.

Exception happens in python-ovs, in __getattr__ which will show the "transaction view" of a Row. So if a row is modified in a transaction by removing a field from a dict, it won't show up when you access Row.column[field]. It looks like a field is moved from a dict in the transaction (it's in "removes") but the field is already no longer in the data. Still investigating.

Comment 14 Eran Kuris 2018-11-01 11:53:12 UTC
Fix verified : 
OpenStack/14.0-RHEL-7/2018-10-30.6
openvswitch2.10-2.10.0-21.el7fdn.x86_64

Comment 17 errata-xmlrpc 2019-01-11 11:53:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045