Bug 1754566

Summary: [OSP16.0.z] DiskNotFound raised during update_available_resource
Product: Red Hat OpenStack Reporter: Candido Campos <ccamposr>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.0 (Train)CC: dasmith, eglynn, jhakimra, kchamart, lyarwood, mgarciac, sbauza, sgordon, vromanso
Target Milestone: ---Keywords: Patch, Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-20.1.2-0.20200226215449.40c288b.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1810440 (view as bug list) Environment:
Last Closed: 2020-05-14 12:10:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1810440, 1810442    
Attachments:
Description Flags
logs none

Description Candido Campos 2019-09-23 15:12:49 UTC 
Description of problem:

Executing neutron tempest test, this error is reported in the log several times and seem be the same problem that:

https://bugs.launchpad.net/nova/+bug/1774249

Error in compute-log:

261-bfd6-8f3f6b560ca5] Updating port 589b13cd-608b-4da6-aa6a-0d2c5d861d9b with attributes {'binding:host_id': 'compute-1.redhat.local', 'device_owner': 'compute:nova'}
2019-09-21 22:31:55.696 9 WARNING nova.compute.manager [req-dc3da9e5-0748-4a1f-b0d8-41bd5e7f68c0 310b28ebcb344cda868ae83c788919ce 7ad20b0cd68f4e84a8ad69e391d753c1 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Received unexpected event network-vif-unplugged-589b13cd-608b-4da6-aa6a-0d2c5d861d9b for instance with vm_state active and task_state resize_migrated.
2019-09-21 22:31:57.032 9 INFO nova.virt.libvirt.driver [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Creating image
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager [req-1463c7df-fb3a-4bd0-b03f-3efe2ce6c3ba - - - - -] Error updating resources for node compute-1.redhat.local.: nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager Traceback (most recent call last):
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     startup=startup)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 728, in update_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7117, in get_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8715, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     err_ctxt.reraise = False
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     self.force_reraise()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise value
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8656, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     config, block_device_info)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8558, in _get_instance_disk_info_from_config
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     qemu_img_info = disk_api.get_disk_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/disk/api.py", line 97, in get_disk_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     return images.qemu_img_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/images.py", line 58, in qemu_img_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise exception.DiskNotFound(location=path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager
2019-09-21 22:31:57.651 9 INFO os_vif [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:ac:3a:d2,bridge_name='qbr589b13cd-60',has_traffic_filtering=True,id=589b13cd-608b-4da6-aa6a-0d2c5d861d9b,network=Network(fe8d8f6c-96d4-4632-ba25-b03f3ca5e28f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap589b13cd-60')
2019-09-21 22:31:58.200 9 INFO nova.compute.manager [req-586f701a-62da-438d-b7cf-31bcaf8704f2 - - - - -] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] VM Resumed (Lifecycle Ev
Comment 1 Candido Campos 2019-09-23 15:15:58 UTC 
Created attachment 1618245 [details]
logs
Comment 2 Matthew Booth 2019-09-27 13:35:25 UTC 
Eurgh. This looks like a race I actually pointed out in review:

https://review.opendev.org/#/c/571410/7/nova/virt/libvirt/driver.py

but which wasn't fixed before the final patch. We should consider taking the fix back to OSP13.
Comment 10 errata-xmlrpc 2020-05-14 12:10:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2154