Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1575130

Summary:	Cinder's image volume cache sync lock prevents parallel image downloading
Product:	Red Hat OpenStack	Reporter:	Alan Bishop <abishop>
Component:	openstack-cinder	Assignee:	Alan Bishop <abishop>
Status:	CLOSED ERRATA	QA Contact:	Avi Avraham <aavraham>
Severity:	medium	Docs Contact:	Kim Nylander <knylande>
Priority:	medium
Version:	10.0 (Newton)	CC:	aavraham, abishop, cschwede, eharney, jamsmith, knylande, nchandek, srevivo, tshefi
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	12.0 (Pike)
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	openstack-cinder-11.1.0-12.el7ost	Doc Type:	Bug Fix
Doc Text:	Simultaneous requests to create a volume from an image now run in parallel when the volume image cache is disabled. When the volume image cache is enabled, locking is minimized to ensure only a single entry is created in the cache. Previously, the scope of the lock was too broad, causing simultaneous volume creation requests to compete for the lock, even when the image cache was not enabled. As a result, simultaneous requests to create a volume from an image were serialized, and not run in parallel.	Story Points:	---
Clone Of:	1572220
Clones:	1575720 (view as bug list)		Environment:
Last Closed:	2018-08-20 12:46:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1572220
Bug Blocks:	1575720, 1575758

Description Alan Bishop 2018-05-04 20:18:03 UTC

+++ This bug was initially created as a clone of Bug #1572220 +++

+++ This bug was initially created as a clone of Bug #1555188 +++

A synchronization lock was added to fix a problem that allowed multiple redundant entries in the image volume cache [1],[2].

[1] https://bugs.launchpad.net/cinder/+bug/1649636
[2] https://review.openstack.org/446590

Unfortunately, the scope of the lock is too broad and impacts performance in a couple of scenarios.

1) When the cache is not enabled, simultaneous attempts to download the image from glance are serialized.
2) Even when the volume image is cached, simultaneous requests to create a volume from the image are serialized.

The scope of the lock needs to be reduced to only serialize the portion of the process that results in an entry being added to the cache.

--- Additional comment from Alan Bishop on 2018-04-26 08:39:02 EDT ---

Patch has merged upstream, and needs to be backported to OSP.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-04-26 08:39:08 EDT ---

This bugzilla has been removed from the Target Release until it has been release planned with 3-acks provided.

--- Additional comment from Jon Schlueter on 2018-05-02 12:15:29 EDT ---

raising needinfo? to rhos-flags@ asking for acks to get POST Bug to have proper acks and release flag before it lands.  If it doesn't have necessary bits or can't be acked bug might need to be moved off of POST until it's clear what is needed to move the bug forward.

--- Additional comment from Alan Bishop on 2018-05-04 16:13:51 EDT ---

The upstream patch merged in time to be included in the latest OSP-13 puddle.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-05-04 16:14:01 EDT ---

This bugzilla has been removed from the Target Release until it has been release planned with 3-acks provided.

Comment 5 Tzach Shefi 2018-07-02 10:30:41 UTC

Verified on:
openstack-cinder-11.1.0-14.el7ost.noarch

With->  image_volume_cache_enabled = false
Uploaded cloud image, simultaneously created two volumes from image, looking good, both volumes created in parallel, as expected. 

Now with image cache enable
Set image_volume_cache_enabled = true
cinder_internal_tenant_project_id = 9c65f22c58f145f8b9e8662b54051489
cinder_internal_tenant_user_id = b5a521e2d5cb49ba9dfddd73d6221207

restart Cinder 

Create 2 volumes from same image.
See the same cached image reference being reused. 


/var/log/cinder/volume.log:2018-07-02 10:28:28.663 677217 DEBUG cinder.image.image_utils [req-1c280dbe-9341-46db-b9ac-683e2877601e b5a521e2d5cb49ba9dfddd73d6221207 9c65f22c58f145f8b9e8662b54051489 - default default] Temporary image 5188c3f9-cf6e-4dfa-997b-006129b2b384 is fetched for user b5a521e2d5cb49ba9dfddd73d6221207. fetch /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:672


Looking fine.

Comment 7 errata-xmlrpc 2018-08-20 12:46:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2516