Bug 1572220

Summary:	Cinder's image volume cache sync lock prevents parallel image downloading
Product:	Red Hat OpenStack	Reporter:	Alan Bishop <abishop>
Component:	openstack-cinder	Assignee:	Alan Bishop <abishop>
Status:	CLOSED ERRATA	QA Contact:	Avi Avraham <aavraham>
Severity:	high	Docs Contact:	Kim Nylander <knylande>
Priority:	medium
Version:	10.0 (Newton)	CC:	aavraham, abishop, cschwede, eharney, knylande, nchandek, srevivo, tshefi
Target Milestone:	beta	Keywords:	Triaged
Target Release:	13.0 (Queens)
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost	Doc Type:	Bug Fix
Doc Text:	The Block Storage service (cinder) uses a synchronization lock to prevent duplicate entries in the volume image cache. The scope of the lock was too broad and caused simultaneous requests to create a volume from an image to compete for the lock, even when the image cache was not enabled. These simultaneous requests to create a volume from an image would be serialized and not run in parallel. As a result, the synchronization lock has been updated to minimize the scope of the lock and to take effect only when the volume image cache is enabled. Now, simultaneous requests to create a volume from an image run in parallel when the volume image cache is disabled. When the volume image cache is enabled, locking is minimized to ensure only a single entry is created in the cache.	Story Points:	---
Clone Of:	1555188
Clones:	1575130 (view as bug list)		Environment:
Last Closed:	2018-06-27 13:53:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1575130, 1575720, 1575758

Description Alan Bishop 2018-04-26 12:38:00 UTC

+++ This bug was initially created as a clone of Bug #1555188 +++

A synchronization lock was added to fix a problem that allowed multiple redundant entries in the image volume cache [1],[2].

[1] https://bugs.launchpad.net/cinder/+bug/1649636
[2] https://review.openstack.org/446590

Unfortunately, the scope of the lock is too broad and impacts performance in a couple of scenarios.

1) When the cache is not enabled, simultaneous attempts to download the image from glance are serialized.
2) Even when the volume image is cached, simultaneous requests to create a volume from the image are serialized.

The scope of the lock needs to be reduced to only serialize the portion of the process that results in an entry being added to the cache.

Comment 1 Alan Bishop 2018-04-26 12:39:02 UTC

Patch has merged upstream, and needs to be backported to OSP.

Comment 4 Alan Bishop 2018-05-04 20:13:51 UTC

The upstream patch merged in time to be included in the latest OSP-13 puddle.

Comment 9 Tzach Shefi 2018-05-28 07:44:34 UTC

Verified on:
openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost.noarch

With->  image_volume_cache_enabled = false
Uploaded cloud image, simultaneously created two volumes from image, looking good, both volumes created in parallel, as expected. 

Now with image cache enable
Set image_volume_cache_enabled = true
cinder_internal_tenant_project_id = ea6a84721c25439bbee43448e658942b
cinder_internal_tenant_user_id = 9206a87622184de58a7c00d32eae7bb3
restart dockers

Create 3 volumes from same image.
See the same cached image reference being reused. 

Temporary image fdc45365-c737-4dd4-bc91-006b        227fa6ef is fetched for user 9206a87622184de58a7c00d32eae7bb3. fetch /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:763

Looking fine.

Comment 11 errata-xmlrpc 2018-06-27 13:53:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086