Bug 1572220 - Cinder's image volume cache sync lock prevents parallel image downloading
Summary: Cinder's image volume cache sync lock prevents parallel image downloading
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 10.0 (Newton)
Hardware: All
OS: All
medium
high
Target Milestone: beta
: 13.0 (Queens)
Assignee: Alan Bishop
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks: 1575130 1575720 1575758
TreeView+ depends on / blocked
 
Reported: 2018-04-26 12:38 UTC by Alan Bishop
Modified: 2018-06-27 13:55 UTC (History)
9 users (show)

Fixed In Version: openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost
Doc Type: Bug Fix
Doc Text:
The Block Storage service (cinder) uses a synchronization lock to prevent duplicate entries in the volume image cache. The scope of the lock was too broad and caused simultaneous requests to create a volume from an image to compete for the lock, even when the image cache was not enabled. These simultaneous requests to create a volume from an image would be serialized and not run in parallel. As a result, the synchronization lock has been updated to minimize the scope of the lock and to take effect only when the volume image cache is enabled. Now, simultaneous requests to create a volume from an image run in parallel when the volume image cache is disabled. When the volume image cache is enabled, locking is minimized to ensure only a single entry is created in the cache.
Clone Of: 1555188
: 1575130 (view as bug list)
Environment:
Last Closed: 2018-06-27 13:53:50 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1758414 0 None None None 2018-04-26 12:37:59 UTC
OpenStack gerrit 556607 0 None MERGED Reduce scope of the lock for image volume cache 2021-02-04 17:19:34 UTC
Red Hat Product Errata RHEA-2018:2086 0 None None None 2018-06-27 13:55:07 UTC

Description Alan Bishop 2018-04-26 12:38:00 UTC
+++ This bug was initially created as a clone of Bug #1555188 +++

A synchronization lock was added to fix a problem that allowed multiple redundant entries in the image volume cache [1],[2].

[1] https://bugs.launchpad.net/cinder/+bug/1649636
[2] https://review.openstack.org/446590

Unfortunately, the scope of the lock is too broad and impacts performance in a couple of scenarios.

1) When the cache is not enabled, simultaneous attempts to download the image from glance are serialized.
2) Even when the volume image is cached, simultaneous requests to create a volume from the image are serialized.

The scope of the lock needs to be reduced to only serialize the portion of the process that results in an entry being added to the cache.

Comment 1 Alan Bishop 2018-04-26 12:39:02 UTC
Patch has merged upstream, and needs to be backported to OSP.

Comment 4 Alan Bishop 2018-05-04 20:13:51 UTC
The upstream patch merged in time to be included in the latest OSP-13 puddle.

Comment 9 Tzach Shefi 2018-05-28 07:44:34 UTC
Verified on:
openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost.noarch

With->  image_volume_cache_enabled = false
Uploaded cloud image, simultaneously created two volumes from image, looking good, both volumes created in parallel, as expected. 

Now with image cache enable
Set image_volume_cache_enabled = true
cinder_internal_tenant_project_id = ea6a84721c25439bbee43448e658942b
cinder_internal_tenant_user_id = 9206a87622184de58a7c00d32eae7bb3
restart dockers

Create 3 volumes from same image.
See the same cached image reference being reused. 

Temporary image fdc45365-c737-4dd4-bc91-006b        227fa6ef is fetched for user 9206a87622184de58a7c00d32eae7bb3. fetch /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:763

Looking fine.

Comment 11 errata-xmlrpc 2018-06-27 13:53:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.