Bug 1810442 - [OSP13.0.z] DiskNotFound raised during update_available_resource
Summary: [OSP13.0.z] DiskNotFound raised during update_available_resource
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Lee Yarwood
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
: 1813596 (view as bug list)
Depends On: 1754566 1810440
Blocks: 1381612
TreeView+ depends on / blocked
 
Reported: 2020-03-05 09:14 UTC by Lee Yarwood
Modified: 2023-09-07 22:12 UTC (History)
13 users (show)

Fixed In Version: openstack-nova-17.0.13-4.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1810440
Environment:
Last Closed: 2020-06-24 11:52:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Nova compute log (2.53 MB, text/plain)
2020-03-05 09:28 UTC, Tzach Shefi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 711279 0 None MERGED libvirt: Ignore DiskNotFound during update_available_resource 2021-02-02 07:39:05 UTC
Red Hat Product Errata RHBA-2020:2725 0 None None None 2020-06-24 11:52:48 UTC

Description Lee Yarwood 2020-03-05 09:14:31 UTC
+++ This bug was initially created as a clone of Bug #1810440 +++

+++ This bug was initially created as a clone of Bug #1754566 +++

Description of problem:

Executing neutron tempest test, this error is reported in the log several times and seem be the same problem that:

https://bugs.launchpad.net/nova/+bug/1774249

Error in compute-log:

261-bfd6-8f3f6b560ca5] Updating port 589b13cd-608b-4da6-aa6a-0d2c5d861d9b with attributes {'binding:host_id': 'compute-1.redhat.local', 'device_owner': 'compute:nova'}
2019-09-21 22:31:55.696 9 WARNING nova.compute.manager [req-dc3da9e5-0748-4a1f-b0d8-41bd5e7f68c0 310b28ebcb344cda868ae83c788919ce 7ad20b0cd68f4e84a8ad69e391d753c1 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Received unexpected event network-vif-unplugged-589b13cd-608b-4da6-aa6a-0d2c5d861d9b for instance with vm_state active and task_state resize_migrated.
2019-09-21 22:31:57.032 9 INFO nova.virt.libvirt.driver [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] Creating image
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager [req-1463c7df-fb3a-4bd0-b03f-3efe2ce6c3ba - - - - -] Error updating resources for node compute-1.redhat.local.: nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager Traceback (most recent call last):
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     startup=startup)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 728, in update_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7117, in get_available_resource
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8715, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     err_ctxt.reraise = False
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     self.force_reraise()
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise value
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8656, in _get_disk_over_committed_size_total
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     config, block_device_info)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8558, in _get_instance_disk_info_from_config
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     qemu_img_info = disk_api.get_disk_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/disk/api.py", line 97, in get_disk_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     return images.qemu_img_info(path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/images.py", line 58, in qemu_img_info
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager     raise exception.DiskNotFound(location=path)
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/215fe835-11b9-4f72-a893-5a7186d2af2f/disk
2019-09-21 22:31:57.081 9 ERROR nova.compute.manager
2019-09-21 22:31:57.651 9 INFO os_vif [req-2422c83c-12a8-42ea-8161-33d4213676f4 c915209a830949e698578b416460c7fe 0467bb57250d427fba43af4254f022d4 - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:ac:3a:d2,bridge_name='qbr589b13cd-60',has_traffic_filtering=True,id=589b13cd-608b-4da6-aa6a-0d2c5d861d9b,network=Network(fe8d8f6c-96d4-4632-ba25-b03f3ca5e28f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap589b13cd-60')
2019-09-21 22:31:58.200 9 INFO nova.compute.manager [req-586f701a-62da-438d-b7cf-31bcaf8704f2 - - - - -] [instance: b355ce24-2a74-4261-bfd6-8f3f6b560ca5] VM Resumed (Lifecycle Ev

--- Additional comment from Candido Campos on 2019-09-23 15:15:58 UTC ---



--- Additional comment from Matthew Booth on 2019-09-27 13:35:25 UTC ---

Eurgh. This looks like a race I actually pointed out in review:

https://review.opendev.org/#/c/571410/7/nova/virt/libvirt/driver.py

but which wasn't fixed before the final patch. We should consider taking the fix back to OSP13.

Comment 1 Tzach Shefi 2020-03-05 09:28:25 UTC
Created attachment 1667691 [details]
Nova compute log

Also hit this issue, 
Added my version/log in case it's useful.

version:
rhel7.7
puppet-nova-12.5.0-5.el7ost.noarch
python2-novaclient-10.1.1-1.el7ost.noarch
openstack-nova-compute-17.0.13-2.el7ost.noarch
python-nova-17.0.13-2.el7ost.noarch
openstack-nova-common-17.0.13-2.el7ost.noarch
openstack-nova-migration-17.0.13-2.el7ost.noarch

Lee's input per my log from an email thread: 
This looks like a race between update_available_resource_for_node and
the deletion of the 11657308-a3d2-4484-b7a0-f30d96bb9327 instance.

Comment 2 Lee Yarwood 2020-03-14 20:08:01 UTC
*** Bug 1813596 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2020-06-24 11:52:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2725


Note You need to log in before you can comment on or make changes to this bug.