Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1797493

Summary: RBD image backend tries to flatten images even if they are already flat
Product: Red Hat OpenStack Reporter: Marian Krcmarik <mkrcmari>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: high Docs Contact:
Priority: high    
Version: 16.0 (Train)CC: dasmith, eglynn, jhakimra, kchamart, lyarwood, sbauza, sgordon, stephenfin, vromanso
Target Milestone: z2Keywords: Patch, Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-20.1.2-0.20200225232948.40c288b.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-14 12:10:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1848814    
Attachments:
Description Flags
nova-compute.log none

Description Marian Krcmarik 2020-02-03 09:49:44 UTC 
Description of problem:
I believe (based on log investigation) that we are hitting a bug filed upstream (on EDGE HCI deployment). I am copying upstram bug description:
When [DEFAULT]show_multiple_locations option is not set in glance, and both glance and nova use ceph as their backend, with properly configured accesses, nova will fail with the following exception:

2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [req-8021fd76-d5ab-4a9b-bd17-f5eb4d4faf62 0e96a04f360644818632b7e46fe8d3e7 ac01daacc7424a40b8b464a163902dcb - default default] [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] Instance failed to spawn: rbd.InvalidArgument: [errno 22] error flattening b'fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6_disk'
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] Traceback (most recent call last):
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 5757, in _unshelve_instance
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] block_device_info=block_device_info)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3457, in spawn
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] block_device_info=block_device_info)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3832, in _create_image
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] fallback_from_host)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3923, in _create_and_inject_local_root
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] instance, size, fallback_from_host)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 9267, in _try_fetch_image_cache
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] image.flatten()
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/imagebackend.py", line 983, in flatten
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] self.driver.flatten(self.rbd_name, pool=self.driver.pool)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 290, in flatten
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] vol.flatten()
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rv = execute(f, *args, **kwargs)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] six.reraise(c, e, tb)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] raise value
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rv = meth(*args, **kwargs)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "rbd.pyx", line 2207, in rbd.Image.flatten
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rbd.InvalidArgument: [errno 22] error flattening b'fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6_disk'
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]

During unshelve, when nova fails to determine the parent pool for the image (because glance api does not return rbd image location), it downloads the image through the glance api and puts it into its pool. Such image will be already flat. And nova will try to flatten it again and fail. It might be better to make flatten idempotent and just be a noop for already flat images.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Deploy EDGE HCI env - DCN with compute nodes having ceph storage backend
2. Run tempest tests related to shelve operation

Actual results:
Any (un)shelve operation on SHELVE_OFFLOADED instance fails.

Expected results:
This tests passes on other EDGE deployments without HCI (with ceph backends)

Additional info:
Comment 1 Lee Yarwood 2020-02-07 11:29:06 UTC 
(In reply to Marian Krcmarik from comment #0)
> Description of problem:
> I believe (based on log investigation) that we are hitting a bug filed
> upstream (on EDGE HCI deployment). I am copying upstram bug description:

Please don't do this in future, referencing the upstream bug is enough.

Actual logs from a downstream environment where you are hitting this are much more relevant and useful.

> Steps to Reproduce:
> 1. Deploy EDGE HCI env - DCN with compute nodes having ceph storage backend
> 2. Run tempest tests related to shelve operation

Just to be clear, Edge HCI / DCN results in separate image and vms clusters right?
Comment 2 Marian Krcmarik 2020-02-07 13:24:23 UTC 
Created attachment 1661680 [details]
nova-compute.log
Comment 3 Marian Krcmarik 2020-02-07 13:36:50 UTC 
(In reply to Lee Yarwood from comment #1)
> (In reply to Marian Krcmarik from comment #0)
> > Description of problem:
> > I believe (based on log investigation) that we are hitting a bug filed
> > upstream (on EDGE HCI deployment). I am copying upstram bug description:
> 
> Please don't do this in future, referencing the upstream bug is enough.
> 
> Actual logs from a downstream environment where you are hitting this are
> much more relevant and useful.
I've attached nova-compute log from my env with the error
> 
> > Steps to Reproduce:
> > 1. Deploy EDGE HCI env - DCN with compute nodes having ceph storage backend
> > 2. Run tempest tests related to shelve operation
> 
> Just to be clear, Edge HCI / DCN results in separate image and vms clusters
> right?
I am not sure I understand question but HCI DCN env is a multistack env with specific stacks for Edge sites, Each edge site consists of several computes nodes (3 in my case) on which ceph cluster is deployed along side with nova-compute service. Each site has its compute nodes placed into Nova availability zone, The same goes for cinder volume service.
Comment 11 errata-xmlrpc 2020-05-14 12:10:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2154