Bug 980977

Summary: [RHS-RHOS] If storage becomes unavailable volume actions fail (error_deleting)
Product: Red Hat OpenStack Reporter: Ben Turner <bturner>
Component: openstack-cinderAssignee: Sergey Gotliv <sgotliv>
Status: CLOSED WONTFIX QA Contact: Dafna Ron <dron>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.0CC: bturner, eharney, grajaiya, jkt, sgotliv, vagarwal
Target Milestone: ---   
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
virt rhos cinder integration
Last Closed: 2014-05-29 15:40:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Turner 2013-07-03 17:50:31 UTC
Description of problem:

If the storage back end becomes available during volume actions the status in the webUI gets stuck:

test-vol-BZ "stopping gluster volume while creating."	10GB	Error_Deleting

I had to go into the database and manually remove the entries to fix the web UI:

mysql --host=localhost --user=root --password=blah
show tables;
select volumes;
select * from volumes;
select id,host,display_name from volumes
delete from volumes where display_name='test-vol-BZ';

Version-Release number of selected component (if applicable):

openstack-cinder-2013.1.2-3.el6ost.noarch

How reproducible:

I have hit this a couple of different ways so far:

-Power off all gluster nodes during a volume create
-Stop mounted gluster volume, then delete the cinder volume

Steps to Reproduce:
1.  Create a new volume on a glusterfs backend
2.  On the gluster server run:

# gluster volume stop testvol

3.  Attempt to delete the volume from the webUI while the gluster volume is stopped.

4.  Restart the gluster volume:

# gluster volume start testvol

Actual results:

The volume status hangs at error_deleting.  I get an error message in the webUI saying:

You do not have permission to delete volume: test-vol-BZ

Expected results:

I am able to delete the volume after storage comes back up.

Additional info:

I can see this being problematic when there are connectivity issues and/or if there are split brain issues on the gluster back end.

Comment 1 Eric Harney 2013-07-03 17:54:16 UTC
I believe "cinder force-delete" will allow you to remove the volume rather than manipulating the database.

Comment 5 Ayal Baron 2013-08-06 09:32:07 UTC
(In reply to Eric Harney from comment #1)
> I believe "cinder force-delete" will allow you to remove the volume rather
> than manipulating the database.

Eric, regardless of a workaround of using force-delete, in the least the error message is incorrect.
In addition, what happens if the user issues delete again? would that work?