Bug 2084239

Summary: nova host-evacuation returns erroneous pci addresses and an error: Unable to correlate PCI slot
Product: Red Hat OpenStack Reporter: smooney
Component: openstack-novaAssignee: Balazs Gibizer <bgibizer>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.2 (Train)CC: alifshit, dasmith, eglynn, jhakimra, kchamart, sbauza, sgordon, vromanso
Target Milestone: z5Keywords: Patch, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-20.6.2-2.20230308185148.fc01371.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-04-26 12:18:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1852110    
Bug Blocks:    

Description smooney 2022-05-11 17:45:18 UTC 
This bug was initially created as a copy of Bug #1852110

I am copying this bug because: 



Basically, one host was down, they nova host-evacuate a few times and have multiple errors.
This is the current error:

~~~
Error: Port update failed for port 23e37c39-9ae7-481f-9bcf-7c725f7ae3b4: Unable to correlate PCI slot 0000:3b:00.3
~~~

[1] Full traceback 


Here we can see the targeted pci addresses:
mysql -N -s -h 172.18.0.97 -u root --password=q1w2e3 -D nova -e "select migration_context from instances a left join instance_extra b on a.uuid = b.instance_uuid where a.uuid = '8c3dc669-2032-4d46-9011-7d6bb14fc1c8'"  | sed 's/\\\\/\\/g' | jq -C '."nova_object.data"."new_pci_devices"."nova_object.data".objects[]."nova_object.data" | .address' 
"0000:3b:01.6"
"0000:3b:00.6"
"0000:3b:01.5"
"0000:3b:00.5"
"0000:3b:02.1"
"0000:3b:01.1"
"0000:3b:02.0"
"0000:3b:01.0"
"0000:3b:01.7"
"0000:3b:00.7"

And this is the original pci_addresses:
$ mysql -N -s -h 172.18.0.97 -u root --password=q1w2e3 -D nova -e "select migration_context from instances a left join instance_extra b on a.uuid = b.instance_uuid where a.uuid = '8c3dc669-2032-4d46-9011-7d6bb14fc1c8'"  | sed 's/\\\\/\\/g' | jq -C '."nova_object.data"."old_pci_devices"."nova_object.data".objects[]."nova_object.data" | .address' 
"0000:3b:00.5"
"0000:3b:00.6"
"0000:3b:00.7"
"0000:3b:01.0"
"0000:3b:01.1"
"0000:3b:01.5"
"0000:3b:01.6"
"0000:3b:01.7"
"0000:3b:02.0"
"0000:3b:02.1"

Customer confirmed that they didn't touch that port, except for binding_host, so I'm wondering where that 0000:3b:00.3 comes from. They did multiple attempts at evacuating the host, so that pci address might come from one of those attempt.


Versions:
openstack-nova-common-17.0.12-1.el7ost.noarch

[1]
~~~
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server [req-ca7443aa-1e20-49cb-a531-94f449cae6ff af0945d62ee34837b02cf42df3d7b157 40eb0ee055d343dab188003e2593beb6 - default default] Exception during message handling: PortUpdateFailed: Port update failed for port 23e37c39-9ae7-481f-9bcf-7c725f7ae3b4: Unable to correlate PCI slot 0000:3b:00.3
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1021, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2958, in rebuild_instance
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     migration, request_spec)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3020, in _do_rebuild_instance_with_claim
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     self._do_rebuild_instance(*args, **kwargs)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3114, in _do_rebuild_instance
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     context, instance, self.host, migration)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2819, in setup_instance_network_on_host
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     migration)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2890, in _update_port_binding_for_instance
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server     pci_slot)
2020-06-28 20:12:19.812 8 ERROR oslo_messaging.rpc.server PortUpdateFailed: Port update failed for port 23e37c39-9ae7-481f-9bcf-7c725f7ae3b4: Unable to correlate PCI slot 0000:3b:00.3
~~~
Comment 14 errata-xmlrpc 2023-04-26 12:18:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: Red Hat OpenStack Platform 16.2 (openstack-nova) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1948