Bug 1555188 - Cinder's image volume cache sync lock prevents parallel image downloading
Summary: Cinder's image volume cache sync lock prevents parallel image downloading
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 10.0 (Newton)
Hardware: All
OS: All
medium
high
Target Milestone: z9
: 10.0 (Newton)
Assignee: Alan Bishop
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
: 1625110 (view as bug list)
Depends On:
Blocks: 1575720 1575758
TreeView+ depends on / blocked
 
Reported: 2018-03-14 06:12 UTC by Nilesh
Modified: 2021-12-10 15:52 UTC (History)
9 users (show)

Fixed In Version: openstack-cinder-9.1.4-36.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, Cinder used a synchronization lock to prevent duplicate entries in the volume image cache. The scope of this lock was too broad, and caused simultaneous requests to create a volume from an image to compete for the lock, even when the image cache was not enabled. As a result of this, simultaneous requests to create a volume from an image were serialized, and not run in parallel. With this update, the scope of the synchronization lock is reduced, and takes effect only when the volume image cache is enabled. Simultaneous requests to create a volume from an image run in parallel when the volume image cache is disabled. When the volume image cache is enabled, locking in minimized to ensure only a single entry is created in the cache.
Clone Of:
: 1572220 (view as bug list)
Environment:
Last Closed: 2018-09-17 16:57:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1758414 0 None None None 2018-03-28 13:51:50 UTC
OpenStack gerrit 556607 0 None MERGED Reduce scope of the lock for image volume cache 2020-12-09 09:03:19 UTC
Red Hat Issue Tracker OSP-4976 0 None None None 2021-12-10 15:52:42 UTC
Red Hat Product Errata RHBA-2018:2717 0 None None None 2018-09-17 16:58:17 UTC

Comment 1 Nilesh 2018-03-14 06:27:48 UTC
Maybe this can help us: 

https://docs.openstack.org/glance/pike/admin/cache.html 

The Glance API server may be configured to have an optional local image cache. A local image cache stores a copy of image files, essentially enabling multiple API servers to serve the same image file, resulting in an increase in scalability due to an increased number of endpoints serving an image file.

Comment 2 Eric Harney 2018-03-14 13:31:23 UTC
Just to be clear, is this referring to creating many instances from the same Glance image, in a boot-from-volume scenario?

Comment 3 Nilesh 2018-03-15 01:16:23 UTC
Yes,  is this referring to creating many instances from the same Glance image.

Comment 4 Alan Bishop 2018-03-28 13:53:22 UTC
Upstream patch has already been accepted.

Comment 5 Alan Bishop 2018-04-26 12:29:20 UTC
Updated the description and removed FutureFeature keyword. The issue is actually a negative side effect of the fix for bug #1434499.

Comment 12 Tzach Shefi 2018-08-28 08:29:02 UTC
Verified on:
openstack-cinder-9.1.4-41.el7ost.noarch

Uploaded ~400MB qcow2 image to Glance.
With default
#image_volume_cache_enabled = false

Changed backend to nfs for larger volume capacity image needs a 10G vol. 

Created three 10G volumes all created simultaneously. 

#cinder create 10 --image rhel    * times 3

+--------------------------------------+-------------+------+------+-------------+----------+-------------+
| ID                                   | Status      | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+------+------+-------------+----------+-------------+
| 39e14288-bc2e-4d85-8fef-0e1bb917836c | downloading | -    | 10   | -           | false    |             |
| 768169d1-05a6-4ee1-8346-1aa0ccc2008d | downloading | -    | 10   | -           | false    |             |
| ae980556-e848-4f2f-ac7e-c33cf4bce97c | downloading | -    | 10   | -           | false    |             |

After a short while, all 3 volumes become available in about the same time.


Now with 
Set image_volume_cache_enabled = true
cinder_internal_tenant_project_id = ea6a84721c25439bbee43448e658942b
cinder_internal_tenant_user_id = 9206a87622184de58a7c00d32eae7bb3

Created same three volumes at once. 


+--------------------------------------+-------------+------+------+-------------+----------+-------------+
| ID                                   | Status      | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+------+------+-------------+----------+-------------+
| 005a72e4-e7a7-4598-97e9-647d751e5052 | downloading | -    | 10   | -           | false    |             |
| 7fb57ae5-9286-4bb6-a37c-58b48b72c786 | creating    | -    | 10   | -           | false    |             |
| af4ecf70-ea4c-437a-82a6-df2f9a5275b9 | creating    | -    | 10   | -           | false    |             |
| d865fef1-939b-4f9e-aefb-75c0152635b0 | deleting    | -    | 4    | -           | false    |             |
+--------------------------------------+-------------+------+------+-------------+----------+-------------+

Ignore deleting volume none relevant. 

A few seconds later:

+--------------------------------------+-------------+------+------+-------------+----------+-------------+
| ID                                   | Status      | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+------+------+-------------+----------+-------------+
| 005a72e4-e7a7-4598-97e9-647d751e5052 | downloading | -    | 10   | -           | false    |             |
| 7fb57ae5-9286-4bb6-a37c-58b48b72c786 | downloading | -    | 10   | -           | false    |             |
| af4ecf70-ea4c-437a-82a6-df2f9a5275b9 | downloading | -    | 10   | -           | false    |             |
| d865fef1-939b-4f9e-aefb-75c0152635b0 | deleting    | -    | 4    | -           | false    |             |


A few second after all 3 are available

+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 005a72e4-e7a7-4598-97e9-647d751e5052 | available | -    | 10   | -           | true     |             |
| 7fb57ae5-9286-4bb6-a37c-58b48b72c786 | available | -    | 10   | -           | true     |             |
| af4ecf70-ea4c-437a-82a6-df2f9a5275b9 | available | -    | 10   | -           | true     |             |
| d865fef1-939b-4f9e-aefb-75c0152635b0 | deleting  | -    | 4    | -           | false    |             |


During this time, we can see all volumes used same cached image:
#tailf /var/log/cinder/volume.log  | grep Temporary

2018-08-28 08:22:51.964 520487 DEBUG cinder.image.image_utils [req-28408103-1ca8-48be-9ccc-b460ff716c66 44a57f1bc9f74b3486c15a8d1f9fc313 5b6dd2c520754296a5a4fe0e2bc4f42e - default default] Temporary image a7ac7b17-b9f0-47bc-aeca-6d591cc05791 is fetched for user 44a57f1bc9f74b3486c15a8d1f9fc313. fetch /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:630                                                                                                            
2018-08-28 08:23:06.103 520487 DEBUG cinder.image.image_utils [req-d819c2e3-ca21-4020-b7b2-b6e73ecd4def 44a57f1bc9f74b3486c15a8d1f9fc313 5b6dd2c520754296a5a4fe0e2bc4f42e - default default] Temporary image a7ac7b17-b9f0-47bc-aeca-6d591cc05791 is fetched for user 44a57f1bc9f74b3486c15a8d1f9fc313. fetch /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:630                                                                                                            
2018-08-28 08:23:10.450 520487 DEBUG cinder.image.image_utils [req-58e4de83-f081-48fb-9f47-f2c68d0fdea2 44a57f1bc9f74b3486c15a8d1f9fc313 5b6dd2c520754296a5a4fe0e2bc4f42e - default default] Temporary image a7ac7b17-b9f0-47bc-aeca-6d591cc05791 is fetched for user 44a57f1bc9f74b3486c15a8d1f9fc313. fetch /usr/lib/python2.7/site-pa

OK to verify.

Comment 13 Alan Bishop 2018-09-04 12:18:35 UTC
*** Bug 1625110 has been marked as a duplicate of this bug. ***

Comment 15 Alan Bishop 2018-09-04 12:56:27 UTC
I don't know what you mean by "already suggested" or what info you seek. But I can tell you the fix that allows for parallel downloading of images (regardless of whether the image cache is enabled) will be available for OSP-10 in openstack-cinder-9.1.4-36.el7ost (see "Fixed In Version" at the top of this page).

Comment 17 errata-xmlrpc 2018-09-17 16:57:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2717


Note You need to log in before you can comment on or make changes to this bug.