2121737 – Error while processing VIF ports: RuntimeError: Switch connection timeout

Bug 2121737 - Error while processing VIF ports: RuntimeError: Switch connection timeout [NEEDINFO]

Summary: Error while processing VIF ports: RuntimeError: Switch connection timeout

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	13.0 (Queens)
Hardware:	All
OS:	All
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Slawek Kaplonski
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-08-26 13:21 UTC by Rohini Diwakar
Modified:	2023-07-29 11:52 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-07-24 09:50:10 UTC
Target Upstream Version:
Embargoed:
Flags:	skaplons: needinfo? (rdiwakar)

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-18417	0	None	None	None	2022-08-26 13:38:46 UTC

Description Rohini Diwakar 2022-08-26 13:21:24 UTC

Description of problem:
When creating a VM it fails with the below error
+++
BuildAbortException: Build of instance 18a918d5-21d3-4afc-92b7-45cae80f7f12 aborted: Failed to allocate the network(s), not rescheduling.\n
+++

If we check the openvswitch-agent.log logs on the compute node, we can see the below traceback
+++
2022-08-26 15:34:09.517 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-25194123-8cb4-454f-a83e-51301c48806f - - - - -] Switch connection timeout: RuntimeError: Switch con
nection timeout
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-25194123-8cb4-454f-a83e-51301c48806f - - - - -] Error while processing VIF ports: RuntimeError: Switch con
nection timeout
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", l
ine 2418, in rpc_loop
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", l
ine 1875, in process_network_ports
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info['removed'])
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", l
ine 1786, in treat_devices_removed
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.port_unbound(device)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1095, in port_unbound
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.reclaim_local_vlan(net_uuid)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 843, in reclaim_local_vlan
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     lvid=lvm.vlan)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_phys.py", line 56, in reclaim_local_vlan
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     (_dp, ofp, ofpp) = self._get_dp()
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 69, in _get_dp
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._cached_dpid = new_dpid
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 52, in _get_dp
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     dp = self._get_dp_by_dpid(self._cached_dpid)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 69, in _get_dp_by_dpid
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: Switch connection timeout
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
+++


Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 13.0.16 (Queens)



Actual results:
VM creation fails

Expected results:
VM should build successfully

Comment 2 Slawek Kaplonski 2022-08-29 14:14:57 UTC

You are using native python interfaces to connect to the ovsdb and to configure OpenFlows in the ovs. We know that in python 2.7 those native interface aren't working well with eventlet and because of that we recommend to change it to the older ovs-vsctl and ofctl interfaces. To do that please set in the neutron-openvswitch-agent's config two options:

[ovs]
of_interface = ovs-ofctl
ovsdb_interface = vsctl

After that please restart the neutron-openvswitch-agent.
If there will be still same timeouts after those changes, please provide us new sosreports with DEBUG log enabled in the neutron-openvswitch-agent and nova-compute.

Note You need to log in before you can comment on or make changes to this bug.