Bug 1016798 - GlusterFS: can't create a snapshot when cinder is configured to work with gluster (permissions)
GlusterFS: can't create a snapshot when cinder is configured to work with glu...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
4.0
x86_64 Linux
high Severity high
: rc
: 4.0
Assigned To: Eric Harney
Dafna Ron
storage
:
: 1029947 1031939 (view as bug list)
Depends On:
Blocks: 1016806 1033652 1033714
  Show dependency treegraph
 
Reported: 2013-10-08 13:21 EDT by Dafna Ron
Modified: 2016-04-26 16:25 EDT (History)
9 users (show)

See Also:
Fixed In Version: openstack-cinder-2013.2-5.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, the Block Storage service did not check first if it had the required permissions to write to a GlusterFS share before creating a snapshot. As a result, if the Block Storage service did not have write permissions to a GlusterFS share, any attempts to create a snapshot on the share would fail. No indication would be given to the user of why the attempt failed, and the volume/snapshot data could be left in an inconsistent state. With this fix, the Block Storage service now checks if it has write permissions to a GlusterFS share before creating a snapshot. Any attempt to create a snapshot would fail with the correct notification (before any data is modified) if the Block Storage service does not have write permissions to the share.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 19:26:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log (177.27 KB, text/plain)
2013-10-08 13:21 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1236966 None None None Never
OpenStack gerrit 57542 None None None Never

  None (edit)
Description Dafna Ron 2013-10-08 13:21:43 EDT
Created attachment 809433 [details]
log

Description of problem:

i configured cinder to work with gluster and when I try to create a snapshot I fail with an error. 

Version-Release number of selected component (if applicable):

openstack-cinder-2013.2-0.9.b3.el6ost.noarch
glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64

How reproducible:

100%

Steps to Reproduce:
1. create a volume
2. boot an instance from the volume 
3. create a snapshot 

Actual results:

we fail to create the snapshot 

Expected results:

we should succeed to create the snapshot 

Additional info:


>>>>>>>>>>>>>>>>>
2013-10-08 19:53:26.918 7297 ERROR cinder.openstack.common.rpc.amqp [req-5beb2863-2be2-439f-8356-80f58b0ec440 c02995f25ba44cfab1a3cbd419f045a1 c77235c29fd0431a8e6628ef6d18e07f] Exception during message handling
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 326, in create_snapshot
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error'})
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 317, in create_snapshot
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_snapshot(snapshot_ref)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 349, in create_snapshot
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     new_snap_path)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 438, in _create_qcow2_snap_file
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     self._execute(*command, run_as_root=True)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef.28dd6a1e-9e96-4dc8-8c09-e38504b6ef6d
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Stdout: "Formatting '/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef.28dd6a1e-9e96-4dc8-8c09-e38504b6ef6d', fmt=qcow2 size=1073741824 backing_file='/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef' encryption=off cluster_size=65536 \n"
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Stderr: '/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef.28dd6a1e-9e96-4dc8-8c09-e38504b6ef6d: error while creating qcow2: No such file or directory\n'




https://bugs.launchpad.net/cinder/+bug/1236966
Comment 1 Eric Harney 2013-10-08 13:50:10 EDT
This is probably related to the permissions of the /var/lib/cinder/mnt/4a31... directory.
Comment 2 Eric Harney 2013-10-08 18:01:10 EDT
I suspect we should be ensuring that the volume permissions are configured with something like this, which should ensure the cinder user can write in the directory.

# gluster volume set cinder-vol storage.owner-uid=165
# gluster volume set cinder-vol storage.owner-gid=165
Comment 3 Eric Harney 2013-10-09 15:57:25 EDT
I've confirmed that the above instructions made this work for me.  (Though had to remove the "=" in the command.)
Comment 4 Giulio Fidente 2013-10-10 09:50:32 EDT
"chown -R cinder: /var/lib/cinder/mnt/"

also worked for me, this is something we should automate in packstack maybe?
Comment 6 Haim 2013-10-22 04:10:58 EDT
removing testBlocker since we have a workaround.
Comment 8 Eric Harney 2013-11-01 11:28:47 EDT
I'm planning to add code to Cinder to detect when the share is not writable, so that snapshot operations will fail earlier, rather than putting the qcow2 files/metadata tracking into an inconsistent state.
Comment 9 Eric Harney 2013-11-19 10:11:27 EST
*** Bug 1031939 has been marked as a duplicate of this bug. ***
Comment 10 Eric Harney 2013-11-20 09:54:24 EST
*** Bug 1029947 has been marked as a duplicate of this bug. ***
Comment 12 Dafna Ron 2013-12-10 13:07:22 EST
verified on 
[root@cougar06 ~(keystone_admin)]# rpm -qa |grep cinder 
python-cinderclient-1.0.7-1.el6ost.noarch
openstack-cinder-2013.2-8.el6ost.noarch
python-cinder-2013.2-8.el6ost.noarch
Comment 14 errata-xmlrpc 2013-12-19 19:26:48 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.