Description of problem:

In openstack-manila running kilo release, if snapshot creation fails for some reason, an entry for snapshot is still created with 'error' state. Attempting to delete this invalid entry of snapshot doesn't succeed.

Possible scenarios which could create a snapshot in error state are when,

 - gluster daemon is down in the backend gluster node
 - gluster node is down
 - gluster brick process is down
 - gluster volume is down
 - snapshot created by manila is deleted from backend gluster

While it is okay to create a snapshot with error state, attempting to delete or force-delete should clear the entry from database. Administrator has no means to remove these snapshots in error state.

Error msg from the logs:

GlusterfsException: Failed to identify backing GlusterFS object for snapshot 0a6e89a3-cbc3-4322-bcb7-05282b3b9816 of share 78fea54d-78f6-4d18-b478-4b40ad2b7c30: a single candidate was expected, 0 was found.

CLI output:

# manila snapshot-list
+--------------------------------------+--------------------------------------+--------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+--------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | error | snap-01 | 1 |
+--------------------------------------+--------------------------------------+--------+---------+------------+
# manila snapshot-delete fac9835a-9689-46a0-8ee7-d2f2d12ad486
# manila snapshot-list
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | error_deleting | snap-01 | 1 |
+--------------------------------------+--------------------------------------+----------------+---------+------------+

# manila snapshot-reset-state fac9835a-9689-46a0-8ee7-d2f2d12ad486
# manila snapshot-list
+--------------------------------------+--------------------------------------+-----------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+-----------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | available | snap-01 | 1 |
+--------------------------------------+--------------------------------------+-----------+---------+------------+

# manila snapshot-force-delete fac9835a-9689-46a0-8ee7-d2f2d12ad486
# manila snapshot-list
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | error_deleting | snap-01 | 1 |
+--------------------------------------+--------------------------------------+----------------+---------+------------+


Version-Release number of selected component (if applicable):

openstack-manila-2015.1.0-2

How reproducible:

Always

Steps to Reproduce:

1) Try to create a snapshot when volume is down (one of the possible ways to create a snapshot that would end up in error state)
2) A snapshot in error state is created
3) Attempt to delete or force-delete the snapshot

Actual results:

snapshot entry in error state is not deleted

Expected results:

snapshot entry in error state should be deleted

Additional info: