OSP 13, neutron ml2-ovs, DPDK - stop/wait/start of nova instance sometimes deletes tbr and yields Error executing command: RowNotFound: Cannot find Bridge with name=
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat OpenStack Platform 13.0 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2023:3902
Description of problem: After running openstack server stop/start, instances didn't come up and stayed in SHUTOFF state. Issue same as described in BZ-1810451 In nova-compute.log, 2022-01-29 01:24:29.080 62 ERROR vif_plug_ovs.linux_net [-] Unable to execute ['ovs-vsctl', '--timeout=120', '--', 'set', 'interface', u'vhu7003efbf-21', 'mtu_request=9000']. Exception: Unexpected error while running command. Command: ovs-vsctl --timeout=120 -- set interface vhu7003efbf-21 mtu_request=9000 Exit code: 1 Stdout: u'' Stderr: u'ovs-vsctl: no row "vhu7003efbf-21" in table Interface\n' 2022-01-29 01:24:29.082 8 ERROR os_vif [req-be2864cb-073c-4e8a-88ad-d07c2d28e094 7223b4f08dc7459d8563318e17a8019f 541dedf4c78243cf9d1c624a3427c91f - default default] Failed to plug vif VIFVHostUser(active=True,address=fa:16:3e:a2:4b:5d,has_traffic_filtering=False,id=7003efbf-2185-412d-b43b-714592b7212b,mode='server',network=Network(0fde5720-ef2b-4038-8739-e7ede246e192),path='/var/lib/vhost_sockets/vhu7003efbf-21',plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='vhu7003efbf-21'): AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', 'set', 'interface', u'vhu7003efbf-21', 'mtu_request=9000'] 2022-01-29 01:24:29.082 8 ERROR os_vif Traceback (most recent call last): 2022-01-29 01:24:29.082 8 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/init.py", line 77, in plug 2022-01-29 01:24:29.082 8 ERROR os_vif plugin.plug(vif, instance_info) 2022-01-29 01:24:29.082 8 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 213, in plug 2022-01-29 01:24:29.082 8 ERROR os_vif self._plug_vhostuser(vif, instance_info) 2022-01-29 01:24:29.082 8 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 148, in _plug_vhostuser 2022-01-29 01:24:29.082 8 ERROR os_vif vif, vif_name, instance_info, *args) 2022-01-29 01:24:29.082 8 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 119, in _create_vif_port 2022-01-29 01:24:29.082 8 ERROR os_vif *kwargs) 2022-01-29 01:24:29.082 8 ERROR os_vif File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap 2022-01-29 01:24:29.082 8 ERROR os_vif return self.channel.remote_call(name, args, kwargs) 2022-01-29 01:24:29.082 8 ERROR os_vif File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2022-01-29 01:24:29.082 8 ERROR os_vif raise exc_type(*result[2]) 2022-01-29 01:24:29.082 8 ERROR os_vif AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', 'set', 'interface', u'vhu7003efbf-21', 'mtu_request=9000'] 2022-01-29 01:24:29.082 8 ERROR os_vif 2022-01-29 01:24:29.085 8 ERROR nova.virt.libvirt.driver [req-be2864cb-073c-4e8a-88ad-d07c2d28e094 7223b4f08dc7459d8563318e17a8019f 541dedf4c78243cf9d1c624a3427c91f - default default] [instance: 3a4ce996-476c-4db3-bdb4-f99dae7dcb72] Failed to start libvirt guest: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFVHostUser(active=True,address=fa:16:3e:a2:4b:5d,has_traffic_filtering=False,id=7003efbf-2185-412d-b43b-714592b7212b,mode='server',network=Network(0fde5720-ef2b-4038-8739-e7ede246e192),path='/var/lib/vhost_sockets/vhu7003efbf-21',plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='vhu7003efbf-21'). Got error: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', 'set', 'interface', u'vhu7003efbf-21', 'mtu_request=9000'] Around the same time in openvswitch-agent.log, 2022-01-29 01:24:25.018 409276 DEBUG neutron.agent.linux.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: {"data":[["aef86567-183 5-43e9-ad53-4bc70ffef38e","delete","vhu7003efbf-21",1,["map",[["attached-mac","fa:16:3e:a2:4b:5d"],["bridge_name","tbr-6fa74ed9-4"],["iface-id","7003efbf-2185-412d-b43b-714592b7212b"],["iface-status","active"],[ "subport_ids","["1a892b9d-5bed-43ad-a13e-f85b9c3c9b23"]"],["trunk_id","6fa74ed9-4e10-4ed8-a9ab-25fe0a8e1f98"],["vm-uuid","3a4ce996-476c-4db3-bdb4-f99dae7dcb72"]]]]],"headings":["row","action","name","ofport"," external_ids"]} _read_stdout /usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py:260 . . 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: RowNotFound: Cannot find Bridge with name=tbr-6fa74ed9-4 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command self.run_idl(None) 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/schema/open_vswitch/commands.py", line 335, in run_idl 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command br = idlutils.row_by_value(self.api.idl, 'Bridge', 'name', self.bridge) 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 63, in row_by_value 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match) 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command RowNotFound: Cannot find Bridge with name=tbr-6fa74ed9-4 2022-01-29 01:24:34.586 409276 ERROR ovsdbapp.backend.ovs_idl.command 2022-01-29 01:24:34.587 409276 ERROR neutron.services.trunk.drivers.openvswitch.agent.ovsdb_handler [-] Cannot obtain interface list for bridge tbr-6fa74ed9-4: Cannot find Bridge with name=tbr-6fa74ed9-4: RowNotFound: Cannot find Bridge with name=tbr-6fa74ed9-4 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: RowNotFound: Cannot find Bridge with name=tbr-6fa74ed9-4 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command self.run_idl(None) 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/schema/open_vswitch/commands.py", line 325, in run_idl 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command br = idlutils.row_by_value(self.api.idl, 'Bridge', 'name', self.bridge) 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 63, in row_by_value 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match) 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command RowNotFound: Cannot find Bridge with name=tbr-6fa74ed9-4 2022-01-29 01:24:34.588 409276 ERROR ovsdbapp.backend.ovs_idl.command 2022-01-29 01:24:34.588 409276 ERROR neutron.services.trunk.drivers.openvswitch.agent.ovsdb_handler [-] Failed to store metadata for trunk 6fa74ed9-4e10-4ed8-a9ab-25fe0a8e1f98: Cannot find Bridge with name=tbr-6fa74ed9-4: RowNotFound: Cannot find Bridge with name=tbr-6fa74ed9-4 2022-01-29 01:24:34.588 409276 DEBUG neutron.services.trunk.drivers.openvswitch.agent.trunk_manager [-] Trunk bridge with ID 6fa74ed9-4e10-4ed8-a9ab-25fe0a8e1f98 does not exist. remove_trunk /usr/lib/python2.7/site-packages/neutron/services/trunk/drivers/openvswitch/agent/trunk_manager.py:231 2022-01-29 01:24:34.589 409276 DEBUG neutron.services.trunk.drivers.openvswitch.agent.ovsdb_handler [-] Deleted resources associated to trunk: 6fa74ed9-4e10-4ed8-a9ab-25fe0a8e1f98 handle_trunk_remove /usr/lib/python2.7/site-packages/neutron/services/trunk/drivers/openvswitch/agent/ovsdb_handler.py:241 Version-Release number of selected component (if applicable): RHOSP13 Additional info: $ rpm -qa | grep openstack-neutron openstack-neutron-l2gw-agent-12.0.2-0.20200729125107.fe947e3.el7ost.noarch openstack-neutron-metering-agent-12.1.1-35.1.el7ost.noarch openstack-neutron-common-12.1.1-35.1.el7ost.noarch openstack-neutron-ml2-12.1.1-35.1.el7ost.noarch openstack-neutron-sriov-nic-agent-12.1.1-35.1.el7ost.noarch openstack-neutron-12.1.1-35.1.el7ost.noarch openstack-neutron-lbaas-12.0.1-0.20190803015156.b86fcef.el7ost.noarch openstack-neutron-linuxbridge-12.1.1-35.1.el7ost.noarch openstack-neutron-lbaas-ui-4.0.1-0.20200422165059.16919a9.el7ost.noarch openstack-neutron-openvswitch-12.1.1-35.1.el7ost.noarch