Bug 2121737

Summary:	Error while processing VIF ports: RuntimeError: Switch connection timeout
Product:	Red Hat OpenStack	Reporter:	Rohini Diwakar <rdiwakar>
Component:	openstack-neutron	Assignee:	Slawek Kaplonski <skaplons>
Status:	CLOSED NOTABUG	QA Contact:	Eran Kuris <ekuris>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	chrisw, jlibosva, scohen, skaplons
Target Milestone:	---	Flags:	skaplons: needinfo? (rdiwakar)
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-24 09:50:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rohini Diwakar 2022-08-26 13:21:24 UTC

Description of problem:
When creating a VM it fails with the below error
+++
BuildAbortException: Build of instance 18a918d5-21d3-4afc-92b7-45cae80f7f12 aborted: Failed to allocate the network(s), not rescheduling.\n
+++

If we check the openvswitch-agent.log logs on the compute node, we can see the below traceback
+++
2022-08-26 15:34:09.517 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-25194123-8cb4-454f-a83e-51301c48806f - - - - -] Switch connection timeout: RuntimeError: Switch con
nection timeout
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-25194123-8cb4-454f-a83e-51301c48806f - - - - -] Error while processing VIF ports: RuntimeError: Switch con
nection timeout
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", l
ine 2418, in rpc_loop
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", l
ine 1875, in process_network_ports
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info['removed'])
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", l
ine 1786, in treat_devices_removed
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.port_unbound(device)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1095, in port_unbound
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.reclaim_local_vlan(net_uuid)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 843, in reclaim_local_vlan
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     lvid=lvm.vlan)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_phys.py", line 56, in reclaim_local_vlan
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     (_dp, ofp, ofpp) = self._get_dp()
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 69, in _get_dp
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._cached_dpid = new_dpid
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 52, in _get_dp
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     dp = self._get_dp_by_dpid(self._cached_dpid)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 69, in _get_dp_by_dpid
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: Switch connection timeout
2022-08-26 15:34:09.518 35318 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
+++


Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 13.0.16 (Queens)



Actual results:
VM creation fails

Expected results:
VM should build successfully

Comment 2 Slawek Kaplonski 2022-08-29 14:14:57 UTC

You are using native python interfaces to connect to the ovsdb and to configure OpenFlows in the ovs. We know that in python 2.7 those native interface aren't working well with eventlet and because of that we recommend to change it to the older ovs-vsctl and ofctl interfaces. To do that please set in the neutron-openvswitch-agent's config two options:

[ovs]
of_interface = ovs-ofctl
ovsdb_interface = vsctl

After that please restart the neutron-openvswitch-agent.
If there will be still same timeouts after those changes, please provide us new sosreports with DEBUG log enabled in the neutron-openvswitch-agent and nova-compute.