2215936 – [OVN migration] VM's with VF's are unable to spawn post migration

Bug 2215936 - [OVN migration] VM's with VF's are unable to spawn post migration

Summary: [OVN migration] VM's with VF's are unable to spawn post migration

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	17.1 (Wallaby)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	ga
Target Release:	17.1
Assignee:	Jakub Libosvar
QA Contact:	Ella Shulman
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-06-19 12:46 UTC by Ella Shulman
Modified:	2023-08-16 01:16 UTC (History)
CC List:	10 users (show)
Fixed In Version:	openstack-neutron-18.6.1-1.20230518200969.el9ost
Doc Type:	Bug Fix
Doc Text:	Before this update, creating an instance with virtual functions (VF) could fail in an environment that had been migrated from ML2/OVS with SR-IOV to ML2/OVN. You can now create instances with VFs after migration.
Clone Of:
Environment:
Last Closed:	2023-08-16 01:15:43 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-25884	0	None	None	None	2023-06-19 12:49:17 UTC
Red Hat Product Errata	RHEA-2023:4577	0	None	None	None	2023-08-16 01:16:19 UTC

Description Ella Shulman 2023-06-19 12:46:45 UTC

Description of problem:
Unable to spawn VMs with VF post ovn migration due to a binding error

/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:00.269 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] 
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [req-b10ce14b-1e0f-4793-a69e-cfcdfa8006f6 83a2ee5457ff4b0d968bc3496857aad7 f26f33d7d37947838e81d15a9a731a16 - default default] [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] Failed to build and run instance: nova.exception.PortBindingFailed: Binding failed for port a0a5123b-8296-426a-bacc-adb1acee36d5, please check neutron logs for more information.
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2414, in _build_and_run_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self.driver.spawn(context, instance, image_meta,
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4284, in spawn
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     xml = self._get_guest_xml(context, instance, network_info,
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7170, in _get_guest_xml
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     network_info_str = str(network_info)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/model.py", line 594, in __str__
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return self._sync_wrapper(fn, *args, **kwargs)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/model.py", line 577, in _sync_wrapper
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self.wait()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/model.py", line 609, in wait
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self[:] = self._gt.wait()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/greenthread.py", line 181, in wait
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return self._exit_event.wait()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     result = hub.switch()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return self.greenlet.switch()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/greenthread.py", line 221, in main
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     result = function(*args, **kwargs)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/utils.py", line 662, in context_wrapper
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return func(*args, **kwargs)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1785, in _allocate_network_async
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     raise e
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1764, in _allocate_network_async
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     nwinfo = self.network_api.allocate_for_instance(
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 1096, in allocate_for_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     created_port_ids = self._update_ports_for_instance(
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 1233, in _update_ports_for_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     vif.destroy()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self.force_reraise()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     raise self.value
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 1202, in _update_ports_for_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     updated_port = self._update_port(
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 586, in _update_port
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     _ensure_no_port_binding_failure(port)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 300, in _ensure_no_port_binding_failure
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     raise exception.PortBindingFailed(port_id=port['id'])
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] nova.exception.PortBindingFailed: Binding failed for port a0a5123b-8296-426a-bacc-adb1acee36d5, please check neutron logs for more information.

Version-Release number of selected component (if applicable):
17.1

How reproducible:
100%

Steps to Reproduce:
1. deploy 17.1 overcloud with SR-IOV configuration enabled and OVS
2. run OVS -> OVN migration procedure 
3. create a VM with VFs

Actual results:
VM creation failed

Expected results:
VM create properly with VF

Additional info:
PFs are created properly

Comment 2 Jakub Libosvar 2023-06-19 16:58:51 UTC

Thanks Ella for providing the environment. The reason for the failure is that sriov agent container is deleted by the migration script because it considers it a neutron agent.

Comment 22 errata-xmlrpc 2023-08-16 01:15:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577

Note You need to log in before you can comment on or make changes to this bug.