Bug 1810442

Summary: [OSP13.0.z] DiskNotFound raised during update_available_resource
Product: Red Hat OpenStack Reporter: Lee Yarwood <lyarwood>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: astupnik, ccamposr, dasmith, ebarrera, eglynn, jhakimra, kchamart, lyarwood, molasaga, nova-maint, sbauza, sgordon, vromanso
Target Milestone: ---Keywords: Patch, Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-17.0.13-4.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1810440 Environment:
Last Closed: 2020-06-24 11:52:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1754566, 1810440    
Bug Blocks: 1381612    
Attachments:
Description Flags
Nova compute log none

Description Lee Yarwood 2020-03-05 09:14:31 UTC 
+++ This bug was initially created as a clone of Bug #1810440 +++

+++ This bug was initially created as a clone of Bug #1754566 +++

Description of problem:

Executing neutron tempest test, this error is reported in the log several times and seem be the same problem that:

https://bugs.launchpad.net/nova/+bug/1774249

Error in compute-log:

261-bfd6-8f3f6b560ca5] Updating port 589b13cd-608b-4da6-aa6a-0d2c5d861d9b with attributes {'binding:host_id': 'compute-1.redhat.local', 'device_owner': 'compute:nova'}
2019-09-21 22:31:55.696 9 WARNING nova.compute.manager [req-dc3da9e5-0748-4a1f-b0d8-41bd5e7f68c0 310b28ebcb344cda868ae83c788919ce 7ad20b0cd68f4e84a8ad69e391d753c1 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Received unexpected event network-vif-unplugged-589b13cd-608b-4da6-aa6a-0d2c5d861d9b for instance with vm_state active and task_state resize_migrated.
2019-09-21 22:31:57.032 9 INFO nova.virt.libvirt.driver [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Creating image
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager [req-1463c7df-fb3a-4bd0-b03f-3efe2ce6c3ba - - - - -] Error updating resources for node compute-1.redhat.local.: nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager Traceback (most recent call last):
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     startup=startup)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 728, in update_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7117, in get_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8715, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     err_ctxt.reraise = False
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     self.force_reraise()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise value
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8656, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     config, block_device_info)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8558, in _get_instance_disk_info_from_config
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     qemu_img_info = disk_api.get_disk_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/disk/api.py", line 97, in get_disk_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     return images.qemu_img_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/images.py", line 58, in qemu_img_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise exception.DiskNotFound(location=path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager
2019-09-21 22:31:57.651 9 INFO os_vif [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:ac:3a:d2,bridge_name='qbr589b13cd-60',has_traffic_filtering=True,id=589b13cd-608b-4da6-aa6a-0d2c5d861d9b,network=Network(fe8d8f6c-96d4-4632-ba25-b03f3ca5e28f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap589b13cd-60')
2019-09-21 22:31:58.200 9 INFO nova.compute.manager [req-586f701a-62da-438d-b7cf-31bcaf8704f2 - - - - -] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] VM Resumed (Lifecycle Ev

--- Additional comment from Candido Campos on 2019-09-23 15:15:58 UTC ---



--- Additional comment from Matthew Booth on 2019-09-27 13:35:25 UTC ---

Eurgh. This looks like a race I actually pointed out in review:

https://review.opendev.org/#/c/571410/7/nova/virt/libvirt/driver.py

but which wasn't fixed before the final patch. We should consider taking the fix back to OSP13.
Comment 1 Tzach Shefi 2020-03-05 09:28:25 UTC 
Created attachment 1667691 [details]
Nova compute log

Also hit this issue, 
Added my version/log in case it's useful.

version:
rhel7.7
puppet-nova-12.5.0-5.el7ost.noarch
python2-novaclient-10.1.1-1.el7ost.noarch
openstack-nova-compute-17.0.13-2.el7ost.noarch
python-nova-17.0.13-2.el7ost.noarch
openstack-nova-common-17.0.13-2.el7ost.noarch
openstack-nova-migration-17.0.13-2.el7ost.noarch

Lee's input per my log from an email thread: 
This looks like a race between update_available_resource_for_node and
the deletion of the 11657308-a3d2-4484-b7a0-f30d96bb9327 instance.
Comment 2 Lee Yarwood 2020-03-14 20:08:01 UTC 
*** Bug 1813596 has been marked as a duplicate of this bug. ***
Comment 17 errata-xmlrpc 2020-06-24 11:52:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2725