1016798 – GlusterFS: can't create a snapshot when cinder is configured to work with gluster (permissions)

Bug 1016798 - GlusterFS: can't create a snapshot when cinder is configured to work with gluster (permissions)

Summary: GlusterFS: can't create a snapshot when cinder is configured to work with glu...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-cinder
Sub Component:
Version:	4.0
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	4.0
Assignee:	Eric Harney
QA Contact:	Dafna Ron
Docs Contact:
URL:
Whiteboard:	storage
Duplicates (2):	1029947 1031939 (view as bug list)
Depends On:
Blocks:	1016806 1033652 1033714
TreeView+	depends on / blocked

Reported:	2013-10-08 17:21 UTC by Dafna Ron
Modified:	2016-04-26 20:25 UTC (History)
CC List:	9 users (show)
Fixed In Version:	openstack-cinder-2013.2-5.el6ost
Doc Type:	Bug Fix
Doc Text:	Previously, the Block Storage service did not check first if it had the required permissions to write to a GlusterFS share before creating a snapshot. As a result, if the Block Storage service did not have write permissions to a GlusterFS share, any attempts to create a snapshot on the share would fail. No indication would be given to the user of why the attempt failed, and the volume/snapshot data could be left in an inconsistent state. With this fix, the Block Storage service now checks if it has write permissions to a GlusterFS share before creating a snapshot. Any attempt to create a snapshot would fail with the correct notification (before any data is modified) if the Block Storage service does not have write permissions to the share.
Clone Of:
Environment:
Last Closed:	2013-12-20 00:26:48 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
log (177.27 KB, text/plain) 2013-10-08 17:21 UTC, Dafna Ron	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1236966	None	None	None	Never
OpenStack gerrit	57542	None	None	None	Never
Red Hat Product Errata	RHEA-2013:1859	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory	2013-12-21 00:01:48 UTC

Description Dafna Ron 2013-10-08 17:21:43 UTC

Created attachment 809433 [details]
log

Description of problem:

i configured cinder to work with gluster and when I try to create a snapshot I fail with an error. 

Version-Release number of selected component (if applicable):

openstack-cinder-2013.2-0.9.b3.el6ost.noarch
glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64

How reproducible:

100%

Steps to Reproduce:
1. create a volume
2. boot an instance from the volume 
3. create a snapshot 

Actual results:

we fail to create the snapshot 

Expected results:

we should succeed to create the snapshot 

Additional info:


>>>>>>>>>>>>>>>>>
2013-10-08 19:53:26.918 7297 ERROR cinder.openstack.common.rpc.amqp [req-5beb2863-2be2-439f-8356-80f58b0ec440 c02995f25ba44cfab1a3cbd419f045a1 c77235c29fd0431a8e6628ef6d18e07f] Exception during message handling
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 326, in create_snapshot
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error'})
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 317, in create_snapshot
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_snapshot(snapshot_ref)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 349, in create_snapshot
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     new_snap_path)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 438, in _create_qcow2_snap_file
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     self._execute(*command, run_as_root=True)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef.28dd6a1e-9e96-4dc8-8c09-e38504b6ef6d
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Stdout: "Formatting '/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef.28dd6a1e-9e96-4dc8-8c09-e38504b6ef6d', fmt=qcow2 size=1073741824 backing_file='/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef' encryption=off cluster_size=65536 \n"
2013-10-08 19:53:26.918 7297 TRACE cinder.openstack.common.rpc.amqp Stderr: '/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364/volume-e78978af-0f46-4caf-948b-218afb5de6ef.28dd6a1e-9e96-4dc8-8c09-e38504b6ef6d: error while creating qcow2: No such file or directory\n'




https://bugs.launchpad.net/cinder/+bug/1236966

Comment 1 Eric Harney 2013-10-08 17:50:10 UTC

This is probably related to the permissions of the /var/lib/cinder/mnt/4a31... directory.

Comment 2 Eric Harney 2013-10-08 22:01:10 UTC

I suspect we should be ensuring that the volume permissions are configured with something like this, which should ensure the cinder user can write in the directory.

# gluster volume set cinder-vol storage.owner-uid=165
# gluster volume set cinder-vol storage.owner-gid=165

Comment 3 Eric Harney 2013-10-09 19:57:25 UTC

I've confirmed that the above instructions made this work for me.  (Though had to remove the "=" in the command.)

Comment 4 Giulio Fidente 2013-10-10 13:50:32 UTC

"chown -R cinder: /var/lib/cinder/mnt/"

also worked for me, this is something we should automate in packstack maybe?

Comment 6 Haim 2013-10-22 08:10:58 UTC

removing testBlocker since we have a workaround.

Comment 8 Eric Harney 2013-11-01 15:28:47 UTC

I'm planning to add code to Cinder to detect when the share is not writable, so that snapshot operations will fail earlier, rather than putting the qcow2 files/metadata tracking into an inconsistent state.

Comment 9 Eric Harney 2013-11-19 15:11:27 UTC

*** Bug 1031939 has been marked as a duplicate of this bug. ***

Comment 10 Eric Harney 2013-11-20 14:54:24 UTC

*** Bug 1029947 has been marked as a duplicate of this bug. ***

Comment 12 Dafna Ron 2013-12-10 18:07:22 UTC

verified on 
[root@cougar06 ~(keystone_admin)]# rpm -qa |grep cinder 
python-cinderclient-1.0.7-1.el6ost.noarch
openstack-cinder-2013.2-8.el6ost.noarch
python-cinder-2013.2-8.el6ost.noarch

Comment 14 errata-xmlrpc 2013-12-20 00:26:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.