Bug 1575130 - Cinder's image volume cache sync lock prevents parallel image downloading
Summary: Cinder's image volume cache sync lock prevents parallel image downloading
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 10.0 (Newton)
Hardware: All
OS: All
medium
medium
Target Milestone: z3
: 12.0 (Pike)
Assignee: Alan Bishop
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
Depends On: 1572220
Blocks: 1575720 1575758
TreeView+ depends on / blocked
 
Reported: 2018-05-04 20:18 UTC by Alan Bishop
Modified: 2018-08-20 12:47 UTC (History)
10 users (show)

Fixed In Version: openstack-cinder-11.1.0-12.el7ost
Doc Type: Bug Fix
Doc Text:
Simultaneous requests to create a volume from an image now run in parallel when the volume image cache is disabled. When the volume image cache is enabled, locking is minimized to ensure only a single entry is created in the cache. Previously, the scope of the lock was too broad, causing simultaneous volume creation requests to compete for the lock, even when the image cache was not enabled. As a result, simultaneous requests to create a volume from an image were serialized, and not run in parallel.
Clone Of: 1572220
: 1575720 (view as bug list)
Environment:
Last Closed: 2018-08-20 12:46:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1758414 0 None None None 2018-05-04 20:18:02 UTC
OpenStack gerrit 558556 0 'None' 'MERGED' 'Reduce scope of the lock for image volume cache' 2019-11-25 04:01:19 UTC
Red Hat Product Errata RHBA-2018:2516 0 None None None 2018-08-20 12:47:05 UTC

Description Alan Bishop 2018-05-04 20:18:03 UTC
+++ This bug was initially created as a clone of Bug #1572220 +++

+++ This bug was initially created as a clone of Bug #1555188 +++

A synchronization lock was added to fix a problem that allowed multiple redundant entries in the image volume cache [1],[2].

[1] https://bugs.launchpad.net/cinder/+bug/1649636
[2] https://review.openstack.org/446590

Unfortunately, the scope of the lock is too broad and impacts performance in a couple of scenarios.

1) When the cache is not enabled, simultaneous attempts to download the image from glance are serialized.
2) Even when the volume image is cached, simultaneous requests to create a volume from the image are serialized.

The scope of the lock needs to be reduced to only serialize the portion of the process that results in an entry being added to the cache.

--- Additional comment from Alan Bishop on 2018-04-26 08:39:02 EDT ---

Patch has merged upstream, and needs to be backported to OSP.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-04-26 08:39:08 EDT ---

This bugzilla has been removed from the Target Release until it has been release planned with 3-acks provided.

--- Additional comment from Jon Schlueter on 2018-05-02 12:15:29 EDT ---

raising needinfo? to rhos-flags@ asking for acks to get POST Bug to have proper acks and release flag before it lands.  If it doesn't have necessary bits or can't be acked bug might need to be moved off of POST until it's clear what is needed to move the bug forward.

--- Additional comment from Alan Bishop on 2018-05-04 16:13:51 EDT ---

The upstream patch merged in time to be included in the latest OSP-13 puddle.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-05-04 16:14:01 EDT ---

This bugzilla has been removed from the Target Release until it has been release planned with 3-acks provided.

Comment 5 Tzach Shefi 2018-07-02 10:30:41 UTC
Verified on:
openstack-cinder-11.1.0-14.el7ost.noarch

With->  image_volume_cache_enabled = false
Uploaded cloud image, simultaneously created two volumes from image, looking good, both volumes created in parallel, as expected. 

Now with image cache enable
Set image_volume_cache_enabled = true
cinder_internal_tenant_project_id = 9c65f22c58f145f8b9e8662b54051489
cinder_internal_tenant_user_id = b5a521e2d5cb49ba9dfddd73d6221207

restart Cinder 

Create 2 volumes from same image.
See the same cached image reference being reused. 


/var/log/cinder/volume.log:2018-07-02 10:28:28.663 677217 DEBUG cinder.image.image_utils [req-1c280dbe-9341-46db-b9ac-683e2877601e b5a521e2d5cb49ba9dfddd73d6221207 9c65f22c58f145f8b9e8662b54051489 - default default] Temporary image 5188c3f9-cf6e-4dfa-997b-006129b2b384 is fetched for user b5a521e2d5cb49ba9dfddd73d6221207. fetch /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:672


Looking fine.

Comment 7 errata-xmlrpc 2018-08-20 12:46:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2516


Note You need to log in before you can comment on or make changes to this bug.