Bug 1179445 - rbd calls block eventlet threads
Summary: rbd calls block eventlet threads
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: unspecified
Hardware: All
OS: Linux
high
high
Target Milestone: beta
: 8.0 (Liberty)
Assignee: Jon Bernard
QA Contact: Yogev Rabl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-06 18:20 UTC by Josh Durgin
Modified: 2016-04-26 13:30 UTC (History)
9 users (show)

Fixed In Version: openstack-cinder-7.0.0-2.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, when Ceph was used as the backing store for Block Storage (cinder), operations such as deleting or flattening a large volume may have blocked other driver threads. Consequently, deleting and flattening threads may have prevented cinder from doing other work until they completed. This fix changes the delete and flattening threads to run in a sub-process, rather than as green threads in the same process. As a result, delete and flattening operations are run in the background so that other cinder operations (such as volume creates and attaches) can run concurrently.
Clone Of:
Environment:
Last Closed: 2016-04-07 21:00:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1401335 0 None None None Never
OpenStack gerrit 145678 0 None None None Never
OpenStack gerrit 175555 0 None None None Never
Red Hat Product Errata RHEA-2016:0603 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 Enhancement Advisory 2016-04-08 00:53:53 UTC

Description Josh Durgin 2015-01-06 18:20:26 UTC
Description of problem:

Some operations in the RBD cinder driver may block for a long time, particularly deleting or flattening a large volume. This can be problematic because it takes up a cinder-volume eventlet thread, and prevents cinder from doing other work if many concurrent flattens or deletes are requested.


Version-Release number of selected component (if applicable):
All

How reproducible:
Always

Steps to Reproduce:
1. Create large volumes
2. Delete them
3. Try another cinder volume operation, like creation

Actual results:
Creation blocks until one of the deletes completes

Expected results:
Creation succeeds quickly


Additional info:

Comment 3 Yogev Rabl 2016-02-15 14:47:26 UTC
verification failed with 
openstack-cinder-7.0.1-6.el7ost.noarch
python-cinder-7.0.1-6.el7ost.noarch

After following the actions, Cinder volumes didn't delete volumes (3 volumes, each 8 TB) after more than 10 minutes.

Comment 4 Josh Durgin 2016-02-15 17:31:46 UTC
(In reply to Yogev Rabl from comment #3)
> verification failed with 
> openstack-cinder-7.0.1-6.el7ost.noarch
> python-cinder-7.0.1-6.el7ost.noarch
> 
> After following the actions, Cinder volumes didn't delete volumes (3
> volumes, each 8 TB) after more than 10 minutes.

For this bug, we're not expecting volume deletion to be fast. With the default settings, deleting an 8TB rbd will be 2million delete ops in ceph (1 per object). We're just expecting it to run in the background and not stop unrelated cinder operations like create or attach other volumes from completing.

Were other operations still getting blocked?

Comment 5 Yogev Rabl 2016-02-16 08:50:12 UTC
No, they weren't. Thus the verification passed.

Comment 7 errata-xmlrpc 2016-04-07 21:00:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html


Note You need to log in before you can comment on or make changes to this bug.