Created attachment 808972 [details] logs Description of problem: This was opened in folsom and fixed. its a regression to Havana. This happened when I failed to mount my gluster share due to issues on the gluster server and tried to create a volume. Version-Release number of selected component (if applicable): [root@cougar06 ~(keystone_admin)]# rpm -qa |grep cinder python-cinder-2013.2-0.9.b3.el6ost.noarch python-cinderclient-1.0.5-1.el6ost.noarch openstack-cinder-2013.2-0.9.b3.el6ost.noarch [root@cougar06 ~(keystone_admin)]# rpm -qa |grep nova openstack-nova-scheduler-2013.2-0.24.rc1.el6ost.noarch openstack-nova-novncproxy-2013.2-0.24.rc1.el6ost.noarch openstack-nova-common-2013.2-0.24.rc1.el6ost.noarch python-novaclient-2.15.0-1.el6ost.noarch python-nova-2013.2-0.24.rc1.el6ost.noarch openstack-nova-console-2013.2-0.24.rc1.el6ost.noarch openstack-nova-conductor-2013.2-0.24.rc1.el6ost.noarch openstack-nova-cert-2013.2-0.24.rc1.el6ost.noarch openstack-nova-compute-2013.2-0.24.rc1.el6ost.noarch openstack-nova-api-2013.2-0.24.rc1.el6ost.noarch openstack-nova-network-2013.2-0.24.rc1.el6ost.noarch How reproducible: Steps to Reproduce: 1. configure cinder to work with gluster 2. hard shut down gluster 3. send a command to create a volume 4. start gluster again Actual results: scheduler reports a problem but the volume is stuck in creating Expected results: if scheduler reports a problem move creating to error Additional info: logs [root@cougar06 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 1560fa00-752b-4d7b-a747-3ef9bf483692 | available | new | 1 | None | True | | | 22c3e84c-1d9b-4a45-9244-06b3ab6c401a | creating | bla | 10 | None | False | | | aadc9c04-17ab-42c4-8bce-c2f63cd287fa | available | image_new | 1 | None | True | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ https://bugs.launchpad.net/nova/+bug/1053931
This belongs to cinder as nova-volume doesn't exist any more. I've been able to reproduce the issue with latest cinder without the need of using glusterfs: 1. kill cinder-volume service: service openstack-cinder-volume stop 2. create a volume before : cinder create 1 3. check the list of volumes: cinder list The volume will be stuck in the creating state forever and cannot be deleted.
(In reply to Dafna Ron from comment #0) > This was opened in folsom and fixed. > its a regression to Havana. What was the behavior in Folsom? It should be possible to delete the volume w/ cinder force-delete.
this is the original bug from launchpad: https://bugs.launchpad.net/nova/+bug/1053931 according to the bug, if the creation fails in the scheduler the status should change to error allowing the user to delete it.
The issue here seems to be the Volume manager needing an initialized driver. Since it was not configured correctly, the driver was disabled on boot. Since this is verified by a python decorator *before* getting into the actual manager method, the failure is not being processed correctly. https://git.openstack.org/cgit/openstack/cinder/tree/cinder/volume/manager.py#n236
*** Bug 1051605 has been marked as a duplicate of this bug. ***
Patch backported upstream to stable/havana
Moving to high since this is quite frustrating from users and leaves the volumes in inconsistent status.
I think that we have a race in the fix. I tested in several ways (iptables, unmount and service stop). I found a problem when I have a cinder stand alone, I stopped the service in the stand alone and then right away created a volume in the controller it seems that because there is a delay in the update of the scheduler the command is sent and get stuck in the scheduler. if we wait a minute and than send the create volume from the controller it will fail right away... so this seems to be a race but I can still reproduce the bug. I think that the bug severity can be lowered because 1. this is now a race 2. we can use the state-reset in cinder to change the status of the volume to available and than delete it. moving back to dev. to reproduce: semi distributed setup: 1. stand alone cinder 2. stand alone glance 3. nova network controller (all other components on this server) 4. one more compute. gluster remote storage configured for cinder 1. on the cinder stand alone host run: '/etc/init.d/openstack-cinder-volume stop' 2. quickly after step 1 (within seconds) run 'cinder create --display-name volume 10' results: cinder volume is stuck in creating. [root@orange-vdsf ~(keystone_admin)]# /etc/init.d/openstack-cinder-volume stop Stopping openstack-cinder-volume: [ OK ] [root@puma31 ~(keystone_admin)]# cinder create --display-name remote-volume-create 10 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-02-18T17:45:12.668313 | | display_description | None | | display_name | remote-volume-create | | id | cec31979-20dd-4d81-b989-a875c26d22cd | | metadata | {} | | size | 10 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@puma31 ~(keystone_admin)]# cinder create --display-name remote-volume-create1 10 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-02-18T17:46:30.287351 | | display_description | None | | display_name | remote-volume-create1 | | id | 01a710f4-fe8f-462b-b2c6-74f539ccb1aa | | metadata | {} | | size | 10 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ As you can see, the first volume is stuck in creating while the second one I created will move to error right away - race? [root@puma31 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | 01a710f4-fe8f-462b-b2c6-74f539ccb1aa | error | remote-volume-create1 | 10 | None | false | | | 9681b5da-975b-4cfe-af5a-a418b19fec81 | available | copy-of-dafna-vol | 12 | None | false | | | cec31979-20dd-4d81-b989-a875c26d22cd | creating | remote-volume-create | 10 | None | false | | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ [root@puma31 ~(keystone_admin)]# logs will be attached.
Created attachment 864672 [details] logs
This sounds like a separate issue. Although it is true that this race can happen, it's quite unlikely and not completely related to the original issue. I cloned the bug #1066955 and I'll move this one to ON_QA again. Thanks a lot Dafna!
moving to verified as agreed with Flavio since the issue is currently reproduced as a race only
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0213.html