Bug 1810440

Summary:	[OSP15.0.z] DiskNotFound raised during update_available_resource
Product:	Red Hat OpenStack	Reporter:	Lee Yarwood <lyarwood>
Component:	openstack-nova	Assignee:	Lee Yarwood <lyarwood>
Status:	CLOSED EOL	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	15.0 (Stein)	CC:	ccamposr, dasmith, eglynn, jhakimra, jschluet, kchamart, lyarwood, nova-maint, sbauza, sgordon, stchen, vromanso
Target Milestone:	---	Keywords:	Patch, Triaged, ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-nova-19.1.1-0.20200221190457.2bc043b.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1754566
Clones:	1810442 (view as bug list)		Environment:
Last Closed:	2020-09-30 19:54:28 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1754566
Bug Blocks:	1810442

Description Lee Yarwood 2020-03-05 09:09:51 UTC

+++ This bug was initially created as a clone of Bug #1754566 +++

Description of problem:

Executing neutron tempest test, this error is reported in the log several times and seem be the same problem that:

https://bugs.launchpad.net/nova/+bug/1774249

Error in compute-log:

261-bfd6-8f3f6b560ca5] Updating port 589b13cd-608b-4da6-aa6a-0d2c5d861d9b with attributes {'binding:host_id': 'compute-1.redhat.local', 'device_owner': 'compute:nova'}
2019-09-21 22:31:55.696 9 WARNING nova.compute.manager [req-dc3da9e5-0748-4a1f-b0d8-41bd5e7f68c0 310b28ebcb344cda868ae83c788919ce 7ad20b0cd68f4e84a8ad69e391d753c1 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Received unexpected event network-vif-unplugged-589b13cd-608b-4da6-aa6a-0d2c5d861d9b for instance with vm_state active and task_state resize_migrated.
2019-09-21 22:31:57.032 9 INFO nova.virt.libvirt.driver [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Creating image
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager [req-1463c7df-fb3a-4bd0-b03f-3efe2ce6c3ba - - - - -] Error updating resources for node compute-1.redhat.local.: nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager Traceback (most recent call last):
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     startup=startup)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 728, in update_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7117, in get_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8715, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     err_ctxt.reraise = False
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     self.force_reraise()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise value
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8656, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     config, block_device_info)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8558, in _get_instance_disk_info_from_config
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     qemu_img_info = disk_api.get_disk_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/disk/api.py", line 97, in get_disk_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     return images.qemu_img_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/images.py", line 58, in qemu_img_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise exception.DiskNotFound(location=path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager
2019-09-21 22:31:57.651 9 INFO os_vif [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:ac:3a:d2,bridge_name='qbr589b13cd-60',has_traffic_filtering=True,id=589b13cd-608b-4da6-aa6a-0d2c5d861d9b,network=Network(fe8d8f6c-96d4-4632-ba25-b03f3ca5e28f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap589b13cd-60')
2019-09-21 22:31:58.200 9 INFO nova.compute.manager [req-586f701a-62da-438d-b7cf-31bcaf8704f2 - - - - -] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] VM Resumed (Lifecycle Ev

--- Additional comment from Candido Campos on 2019-09-23 15:15:58 UTC ---



--- Additional comment from Matthew Booth on 2019-09-27 13:35:25 UTC ---

Eurgh. This looks like a race I actually pointed out in review:

https://review.opendev.org/#/c/571410/7/nova/virt/libvirt/driver.py

but which wasn't fixed before the final patch. We should consider taking the fix back to OSP13.

Comment 4 stchen 2020-09-30 19:54:28 UTC

Closing EOL, OSP 15 has been retired as of Sept 19, 2020