2084239 – nova host-evacuation returns erroneous pci addresses and an error: Unable to correlate PCI slot

Bug 2084239 - nova host-evacuation returns erroneous pci addresses and an error: Unable to correlate PCI slot

Summary: nova host-evacuation returns erroneous pci addresses and an error: Unable to ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	16.2 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	z5
Target Release:	16.2 (Train on RHEL 8.4)
Assignee:	Balazs Gibizer
QA Contact:	OSP DFG:Compute
Docs Contact:
URL:
Whiteboard:
Depends On:	1852110
Blocks:
TreeView+	depends on / blocked

Reported:	2022-05-11 17:45 UTC by smooney
Modified:	2023-04-26 12:18 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openstack-nova-20.6.2-2.20230308185148.fc01371.el8ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-04-26 12:18:09 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1896463	None	None	None	2022-05-11 17:46:54 UTC
OpenStack gerrit	841288	None	NEW	Reproduce bug 1896463 in func env	2022-05-30 15:44:45 UTC
OpenStack gerrit	841444	None	NEW	Set instance host and drop migration under lock	2022-05-30 15:44:47 UTC
Red Hat Issue Tracker	OSP-15182	None	None	None	2022-05-11 17:57:44 UTC
Red Hat Product Errata	RHSA-2023:1948	None	None	None	2023-04-26 12:18:57 UTC

Description smooney 2022-05-11 17:45:18 UTC

This bug was initially created as a copy of Bug #1852110

I am copying this bug because: 



Basically, one host was down, they nova host-evacuate a few times and have multiple errors.
This is the current error:

~~~
Error: Port update failed for port 23e37c39-9ae7-481f-9bcf-7c725f7ae3b4: Unable to correlate PCI slot 0000:3b:00.3
~~~

[1] Full traceback 


Here we can see the targeted pci addresses:
mysql -N -s -h 172.18.0.97 -u root --password=q1w2e3 -D nova -e "select migration_context from instances a left join instance_extra b on a.uuid = b.instance_uuid where a.uuid = '8c3dc669-2032-4d46-9011-7d6bb14fc1c8'"  | sed 's/\\\\/\\/g' | jq -C '."nova_object.data"."new_pci_devices"."nova_object.data".objects[]."nova_object.data" | .address' 
"0000:3b:01.6"
"0000:3b:00.6"
"0000:3b:01.5"
"0000:3b:00.5"
"0000:3b:02.1"
"0000:3b:01.1"
"0000:3b:02.0"
"0000:3b:01.0"
"0000:3b:01.7"
"0000:3b:00.7"

And this is the original pci_addresses:
$ mysql -N -s -h 172.18.0.97 -u root --password=q1w2e3 -D nova -e "select migration_context from instances a left join instance_extra b on a.uuid = b.instance_uuid where a.uuid = '8c3dc669-2032-4d46-9011-7d6bb14fc1c8'"  | sed 's/\\\\/\\/g' | jq -C '."nova_object.data"."old_pci_devices"."nova_object.data".objects[]."nova_object.data" | .address' 
"0000:3b:00.5"
"0000:3b:00.6"
"0000:3b:00.7"
"0000:3b:01.0"
"0000:3b:01.1"
"0000:3b:01.5"
"0000:3b:01.6"
"0000:3b:01.7"
"0000:3b:02.0"
"0000:3b:02.1"

Customer confirmed that they didn't touch that port, except for binding_host, so I'm wondering where that 0000:3b:00.3 comes from. They did multiple attempts at evacuating the host, so that pci address might come from one of those attempt.


Versions:
openstack-nova-common-17.0.12-1.el7ost.noarch

[1]
~~~
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server [req-ca7443aa-1e20-49cb-a531-94f449cae6ff af0945d62ee34837b02cf42df3d7b157 40eb0ee055d343dab188003e2593beb6 - default default] Exception during message handling: PortUpdateFailed: Port update failed for port 23e37c39-9ae7-481f-9bcf-7c725f7ae3b4: Unable to correlate PCI slot 0000:3b:00.3
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1021, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2958, in rebuild_instance
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     migration, request_spec)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3020, in _do_rebuild_instance_with_claim
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self._do_rebuild_instance(*args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3114, in _do_rebuild_instance
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     context, instance, self.host, migration)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2819, in setup_instance_network_on_host
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     migration)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2890, in _update_port_binding_for_instance
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     pci_slot)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server PortUpdateFailed: Port update failed for port 23e37c39-9ae7-481f-9bcf-7c725f7ae3b4: Unable to correlate PCI slot 0000:3b:00.3
~~~

Comment 14 errata-xmlrpc 2023-04-26 12:18:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: Red Hat OpenStack Platform 16.2 (openstack-nova) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1948

Note You need to log in before you can comment on or make changes to this bug.