Bug 2215936

Summary:	[OVN migration] VM's with VF's are unable to spawn post migration
Product:	Red Hat OpenStack	Reporter:	Ella Shulman <eshulman>
Component:	openstack-neutron	Assignee:	Jakub Libosvar <jlibosva>
Status:	CLOSED ERRATA	QA Contact:	Ella Shulman <eshulman>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	17.1 (Wallaby)	CC:	averdagu, bcafarel, chrisw, hakhande, jamsmith, jjung, jlibosva, pgrist, rheslop, scohen
Target Milestone:	ga	Keywords:	Triaged
Target Release:	17.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-18.6.1-1.20230518200969.el9ost	Doc Type:	Bug Fix
Doc Text:	Before this update, creating an instance with virtual functions (VF) could fail in an environment that had been migrated from ML2/OVS with SR-IOV to ML2/OVN. You can now create instances with VFs after migration.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-16 01:15:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ella Shulman 2023-06-19 12:46:45 UTC

Description of problem:
Unable to spawn VMs with VF post ovn migration due to a binding error

/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:00.269 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] 
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [req-b10ce14b-1e0f-4793-a69e-cfcdfa8006f6 83a2ee5457ff4b0d968bc3496857aad7 f26f33d7d37947838e81d15a9a731a16 - default default] [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] Failed to build and run instance: nova.exception.PortBindingFailed: Binding failed for port a0a5123b-8296-426a-bacc-adb1acee36d5, please check neutron logs for more information.
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2414, in _build_and_run_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self.driver.spawn(context, instance, image_meta,
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4284, in spawn
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     xml = self._get_guest_xml(context, instance, network_info,
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7170, in _get_guest_xml
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     network_info_str = str(network_info)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/model.py", line 594, in __str__
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return self._sync_wrapper(fn, *args, **kwargs)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/model.py", line 577, in _sync_wrapper
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self.wait()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/model.py", line 609, in wait
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self[:] = self._gt.wait()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/greenthread.py", line 181, in wait
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return self._exit_event.wait()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     result = hub.switch()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return self.greenlet.switch()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/eventlet/greenthread.py", line 221, in main
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     result = function(*args, **kwargs)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/utils.py", line 662, in context_wrapper
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     return func(*args, **kwargs)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1785, in _allocate_network_async
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     raise e
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1764, in _allocate_network_async
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     nwinfo = self.network_api.allocate_for_instance(
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 1096, in allocate_for_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     created_port_ids = self._update_ports_for_instance(
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 1233, in _update_ports_for_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     vif.destroy()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     self.force_reraise()
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     raise self.value
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 1202, in _update_ports_for_instance
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     updated_port = self._update_port(
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 586, in _update_port
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     _ensure_no_port_binding_failure(port)
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 300, in _ensure_no_port_binding_failure
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120]     raise exception.PortBindingFailed(port_id=port['id'])
/var/log/containers/nova/nova-compute.log.1:2023-06-19 08:46:03.982 2 ERROR nova.compute.manager [instance: ce96f400-9a9f-4729-9c33-6a5d2542e120] nova.exception.PortBindingFailed: Binding failed for port a0a5123b-8296-426a-bacc-adb1acee36d5, please check neutron logs for more information.

Version-Release number of selected component (if applicable):
17.1

How reproducible:
100%

Steps to Reproduce:
1. deploy 17.1 overcloud with SR-IOV configuration enabled and OVS
2. run OVS -> OVN migration procedure 
3. create a VM with VFs

Actual results:
VM creation failed

Expected results:
VM create properly with VF

Additional info:
PFs are created properly

Comment 2 Jakub Libosvar 2023-06-19 16:58:51 UTC

Thanks Ella for providing the environment. The reason for the failure is that sriov agent container is deleted by the migration script because it considers it a neutron agent.

Comment 22 errata-xmlrpc 2023-08-16 01:15:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577