Description of problem:

"neutron agent-list" is blocked, we see traces [1] in Neutron Server logs. It looks like some OVN table contain incorrect entry, but it is hard to understand which one.

I am setting high severity because we need a workaround for production RHOSP deployment. It could be reduced after workaround is applied.

Version-Release number of selected component (if applicable):
16.1.5 GA

How reproducible:
It is unclear which steps caused this state of OVN DB. At this point "neutron agent-list" is blocked.

[1]
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource [req-24b87380-c9bc-4b8c-b30f-0d9dd8f74c23 del del - default default] index failed: No details.: AttributeError: Row instance has no attribute 'hostname'
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource Traceback (most recent call last):
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib64/python3.6/site-packages/ovs/db/idl.py", line 1018, in __getattr__
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     column = self._table.columns[column_name]
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource KeyError: 'hostname'
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource Traceback (most recent call last):
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     self.force_reraise()
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     raise value
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     self.force_reraise()
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     raise value
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     self.force_reraise()
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     raise value
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 369, in index
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     return self._items(request, True, parent_id)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 304, in _items
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     obj_list = obj_getter(request.context, **kwargs)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/mech_driver.py", line 1060, in fn
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     return op(results, new_method(*args, _driver=self, **kwargs))
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/mech_driver.py", line 1132, in get_agents
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     for agent in _driver.agents_from_chassis(ch, update_db).values():
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/mech_driver.py", line 1050, in agents_from_chassis
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     agent_dict[agent.agent_id] = agent.as_dict(alive)
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/networking_ovn/agent/neutron_agent.py", line 51, in as_dict
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     'host': self.chassis.hostname,
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource   File "/usr/lib64/python3.6/site-packages/ovs/db/idl.py", line 1021, in __getattr__
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource     (self.__class__.__name__, column_name))
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource AttributeError: Row instance has no attribute 'hostname'
2021-06-23 12:37:16.373 35 ERROR neutron.api.v2.resource
2021-06-23 12:37:16.375 35 INFO neutron.wsgi [req-24b87380-c9bc-4b8c-b30f-0d9dd8f74c23 del del - default default] 192.168.1.1 "GET /v2.0/agents HTTP/1.1" status: 500  len: 344 time: 0.3377833