Bug 1179445

Summary:	rbd calls block eventlet threads
Product:	Red Hat OpenStack	Reporter:	Josh Durgin <jdurgin>
Component:	openstack-cinder	Assignee:	Jon Bernard <jobernar>
Status:	CLOSED ERRATA	QA Contact:	Yogev Rabl <yrabl>
Severity:	high	Docs Contact:
Priority:	high
Version:	unspecified	CC:	eharney, jschluet, mcsaucy, mlopes, scohen, sgotliv, tbarron, yeylon, yrabl
Target Milestone:	beta
Target Release:	8.0 (Liberty)
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-cinder-7.0.0-2.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, when Ceph was used as the backing store for Block Storage (cinder), operations such as deleting or flattening a large volume may have blocked other driver threads. Consequently, deleting and flattening threads may have prevented cinder from doing other work until they completed. This fix changes the delete and flattening threads to run in a sub-process, rather than as green threads in the same process. As a result, delete and flattening operations are run in the background so that other cinder operations (such as volume creates and attaches) can run concurrently.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-04-07 21:00:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Josh Durgin 2015-01-06 18:20:26 UTC

Description of problem:

Some operations in the RBD cinder driver may block for a long time, particularly deleting or flattening a large volume. This can be problematic because it takes up a cinder-volume eventlet thread, and prevents cinder from doing other work if many concurrent flattens or deletes are requested.


Version-Release number of selected component (if applicable):
All

How reproducible:
Always

Steps to Reproduce:
1. Create large volumes
2. Delete them
3. Try another cinder volume operation, like creation

Actual results:
Creation blocks until one of the deletes completes

Expected results:
Creation succeeds quickly


Additional info:

Comment 3 Yogev Rabl 2016-02-15 14:47:26 UTC

verification failed with 
openstack-cinder-7.0.1-6.el7ost.noarch
python-cinder-7.0.1-6.el7ost.noarch

After following the actions, Cinder volumes didn't delete volumes (3 volumes, each 8 TB) after more than 10 minutes.

Comment 4 Josh Durgin 2016-02-15 17:31:46 UTC

(In reply to Yogev Rabl from comment #3)
> verification failed with 
> openstack-cinder-7.0.1-6.el7ost.noarch
> python-cinder-7.0.1-6.el7ost.noarch
> 
> After following the actions, Cinder volumes didn't delete volumes (3
> volumes, each 8 TB) after more than 10 minutes.

For this bug, we're not expecting volume deletion to be fast. With the default settings, deleting an 8TB rbd will be 2million delete ops in ceph (1 per object). We're just expecting it to run in the background and not stop unrelated cinder operations like create or attach other volumes from completing.

Were other operations still getting blocked?

Comment 5 Yogev Rabl 2016-02-16 08:50:12 UTC

No, they weren't. Thus the verification passed.

Comment 7 errata-xmlrpc 2016-04-07 21:00:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html