Description of problem:
Created a bootable volume from an image and booted an instance. I could create and delete one snapshot. The subsequent snapshots failed with error. Tested on RHEL 6.5

Version-Release number of selected component (if applicable):
glusterfs-fuse-3.4.0.57rhs-1.el6_4.x86_64
fuse-libs-2.8.3-4.el6.x86_64
fuse-2.8.3-4.el6.x86_64
openstack-cinder-2013.2-9.el6ost.noarch

How reproducible: Always


Steps to Reproduce:

1. Create a volume from image:
# cinder create --image-id fbc1d618-f43d-44cd-833b-e5e9d30f8605 10

2. Once the volume becomes available, boot an instance using the volume.
# nova boot --flavor 2 --boot-volume 17339cd6-0464-4dc2-872c-d7d46184fc5f test

3. Create a snapshot of the attached volume.

 # cinder snapshot-create --force True --display-name test 17339cd6-0464-4dc2-872c-d7d46184fc5f
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|      created_at     |      2014-01-22T08:48:23.559706      |
| display_description |                 None                 |
|     display_name    |                 test                 |
|          id         | ba05d4a8-d527-41b8-be4e-5dd33b06ac8e |
|       metadata      |                  {}                  |
|         size        |                  10                  |
|        status       |               creating               |
|      volume_id      | 17339cd6-0464-4dc2-872c-d7d46184fc5f |
+---------------------+--------------------------------------+


# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 17339cd6-0464-4dc2-872c-d7d46184fc5f |   in-use  |     None     |  10  |     None    |   true   | c4ca46df-aed6-45b7-a380-c23f07842df2 |



# cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+
|                  ID                  |              Volume ID               |   Status  | Display Name | Size |
+--------------------------------------+--------------------------------------+-----------+--------------+------+
| ba05d4a8-d527-41b8-be4e-5dd33b06ac8e | 17339cd6-0464-4dc2-872c-d7d46184fc5f | available |     test     |  10  |
+--------------------------------------+--------------------------------------+-----------+--------------+------+


4. Delete this snapshot:

# cinder snapshot-delete ba05d4a8-d527-41b8-be4e-5dd33b06ac8e
# cinder snapshot-list

The snapshot is not listed anymore.

5. Create another snapshot of the same volume:

# cinder snapshot-create --force True --display-name test 17339cd6-0464-4dc2-872c-d7d46184fc5f


Actual results:

The second snapshot creation fails with error:

# cinder snapshot-list
+--------------------------------------+--------------------------------------+--------+--------------+------+
|                  ID                  |              Volume ID               | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+--------+--------------+------+
| 9b243e79-4a51-4832-8c2d-f33ed3ad8f8e | 17339cd6-0464-4dc2-872c-d7d46184fc5f | error  |     test     |  10  |
+--------------------------------------+--------------------------------------+--------+--------------+------+

I see that the first snapshot still exists in the nova/mnt

# ll /var/lib/nova/mnt/4db90e5492997091a102ba6ad764dade
total 17885
-rw-rw-rw-. 1 qemu   qemu   10737418240 Jan 22 14:05 volume-17339cd6-0464-4dc2-872c-d7d46184fc5f
-rw-r--r--. 1 qemu   qemu        197120 Jan 22 14:06 volume-17339cd6-0464-4dc2-872c-d7d46184fc5f.ba05d4a8-d527-41b8-be4e-5dd33b06ac8e
-rw-r--r--. 1 cinder cinder         223 Jan 22 14:06 volume-17339cd6-0464-4dc2-872c-d7d46184fc5f.info


Expected results:

Should be able to create subsequent snapshots.

Additional info:


Logs from cinder/volume.log:

2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_
data
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     **args)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dis
patch
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 809, in wrapper
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 371, in create_snapshot
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error'})
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 362, in create_snapshot
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_snapshot(snapshot_ref)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 389, in create_snapshot
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     new_snap_path)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 478, in _create_qcow2_snap_file
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     self._execute(*command, run_as_root=True)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 143, in execute
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/volumes/4db90e5492997091a102ba6ad764dade/volume-17339cd6-0464-4dc2-872c-d7d46184fc5f.ba05d4a8-d527-41b8-be4e-5dd33b06ac8e /var/lib/cinder/volumes/4db90e5492997091a102ba6ad764dade/volume-17339cd6-0464-4dc2-872c-d7d46184fc5f.9b243e79-4a51-4832-8c2d-f33ed3ad8f8e
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp Stdout: ''
2014-01-22 14:24:10.499 857 TRACE cinder.openstack.common.rpc.amqp Stderr: "Could not open '/var/lib/cinder/volumes/4db90e5492997091a102ba6ad764dade/volume-17339cd6-0464-4dc2-872c-d7d46184fc5f.ba05d4a8-d527-41b8-be4e-5dd33b06ac8e': No such file or directory\n"