Bug 1016224
Summary: | cinder: create volume stuck in creating even though scheduler reports failure | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Dafna Ron <dron> | ||||||
Component: | openstack-cinder | Assignee: | Flavio Percoco <fpercoco> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Dafna Ron <dron> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | urgent | ||||||||
Version: | 4.0 | CC: | abaron, dron, eharney, fpercoco, mlopes, ndipanov, sclewis, scohen, xqueralt, yeylon | ||||||
Target Milestone: | z2 | Keywords: | ZStream | ||||||
Target Release: | 4.0 | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | openstack-cinder-2013.2.1-6.el6ost | Doc Type: | Bug Fix | ||||||
Doc Text: |
Prior to this update, the driver initialization check was done prior to calling the method responsible for processing the RPC request. Consequently, volumes would enter an inconsistent state ('creating' instead of 'error'), which resulted in unusable and stuck volumes.
This issue has been resolved with this update, and the driver check has been moved into the method itself.
|
Story Points: | --- | ||||||
Clone Of: | |||||||||
: | 1051605 1066955 (view as bug list) | Environment: | |||||||
Last Closed: | 2014-03-04 20:12:46 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 977865, 1016216, 1035891, 1051605, 1051606, 1066955 | ||||||||
Attachments: |
|
Description
Dafna Ron
2013-10-07 17:58:04 UTC
This belongs to cinder as nova-volume doesn't exist any more. I've been able to reproduce the issue with latest cinder without the need of using glusterfs: 1. kill cinder-volume service: service openstack-cinder-volume stop 2. create a volume before : cinder create 1 3. check the list of volumes: cinder list The volume will be stuck in the creating state forever and cannot be deleted. (In reply to Dafna Ron from comment #0) > This was opened in folsom and fixed. > its a regression to Havana. What was the behavior in Folsom? It should be possible to delete the volume w/ cinder force-delete. this is the original bug from launchpad: https://bugs.launchpad.net/nova/+bug/1053931 according to the bug, if the creation fails in the scheduler the status should change to error allowing the user to delete it. The issue here seems to be the Volume manager needing an initialized driver. Since it was not configured correctly, the driver was disabled on boot. Since this is verified by a python decorator *before* getting into the actual manager method, the failure is not being processed correctly. https://git.openstack.org/cgit/openstack/cinder/tree/cinder/volume/manager.py#n236 *** Bug 1051605 has been marked as a duplicate of this bug. *** Patch backported upstream to stable/havana Moving to high since this is quite frustrating from users and leaves the volumes in inconsistent status. I think that we have a race in the fix. I tested in several ways (iptables, unmount and service stop). I found a problem when I have a cinder stand alone, I stopped the service in the stand alone and then right away created a volume in the controller it seems that because there is a delay in the update of the scheduler the command is sent and get stuck in the scheduler. if we wait a minute and than send the create volume from the controller it will fail right away... so this seems to be a race but I can still reproduce the bug. I think that the bug severity can be lowered because 1. this is now a race 2. we can use the state-reset in cinder to change the status of the volume to available and than delete it. moving back to dev. to reproduce: semi distributed setup: 1. stand alone cinder 2. stand alone glance 3. nova network controller (all other components on this server) 4. one more compute. gluster remote storage configured for cinder 1. on the cinder stand alone host run: '/etc/init.d/openstack-cinder-volume stop' 2. quickly after step 1 (within seconds) run 'cinder create --display-name volume 10' results: cinder volume is stuck in creating. [root@orange-vdsf ~(keystone_admin)]# /etc/init.d/openstack-cinder-volume stop Stopping openstack-cinder-volume: [ OK ] [root@puma31 ~(keystone_admin)]# cinder create --display-name remote-volume-create 10 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-02-18T17:45:12.668313 | | display_description | None | | display_name | remote-volume-create | | id | cec31979-20dd-4d81-b989-a875c26d22cd | | metadata | {} | | size | 10 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@puma31 ~(keystone_admin)]# cinder create --display-name remote-volume-create1 10 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-02-18T17:46:30.287351 | | display_description | None | | display_name | remote-volume-create1 | | id | 01a710f4-fe8f-462b-b2c6-74f539ccb1aa | | metadata | {} | | size | 10 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ As you can see, the first volume is stuck in creating while the second one I created will move to error right away - race? [root@puma31 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | 01a710f4-fe8f-462b-b2c6-74f539ccb1aa | error | remote-volume-create1 | 10 | None | false | | | 9681b5da-975b-4cfe-af5a-a418b19fec81 | available | copy-of-dafna-vol | 12 | None | false | | | cec31979-20dd-4d81-b989-a875c26d22cd | creating | remote-volume-create | 10 | None | false | | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ [root@puma31 ~(keystone_admin)]# logs will be attached. Created attachment 864672 [details]
logs
This sounds like a separate issue. Although it is true that this race can happen, it's quite unlikely and not completely related to the original issue. I cloned the bug #1066955 and I'll move this one to ON_QA again. Thanks a lot Dafna! moving to verified as agreed with Flavio since the issue is currently reproduced as a race only Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0213.html |