Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1029947

Summary:	GlusterFS, fail to create a new volume with from an old volume
Product:	Red Hat OpenStack	Reporter:	Yogev Rabl <yrabl>
Component:	openstack-cinder	Assignee:	Eric Harney <eharney>
Status:	CLOSED DUPLICATE	QA Contact:	Haim <hateya>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	4.0	CC:	eharney, hateya, sgordon, yeylon, yrabl
Target Milestone:	rc	Keywords:	Regression, Unconfirmed
Target Release:	4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	storage
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-11-20 14:54:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yogev Rabl 2013-11-13 15:24:51 UTC

Description of problem:


A Cinder that has GlusterFS as a back end, fails to create a new volume from an old volume with the command: cinder create --source-volid <old volume id> <volume size> . 
The configuration is: 
Three servers:
1 with Nova.
1 with Nova, Gluster, Ceilometer, Horizon & Keystone.
1 with Cinder. 

the error message is: 

2013-11-13 17:05:26.923 1314 WARNING cinder.taskflow.utils [-] Activating 5 rollbacks due to <cinder.taskflow.utils.FlowFailure object at 0x3308fd0>.
2013-11-13 17:05:27.031 1314 ERROR cinder.volume.flows.create_volume [req-11ba4c67-ac02-430b-85bf-4dfdf0694ca4 81b1ae91395f4f408f6870cb15531ce3 ea0a2ce364fe417aba3aa8d47a377eae] Volume 21834cd6-f192-4bce-96ae-7f787ad2f320: create failed                                                                                                                                                                                              
2013-11-13 17:05:27.032 1314 ERROR cinder.volume.flows.create_volume [req-11ba4c67-ac02-430b-85bf-4dfdf0694ca4 81b1ae91395f4f408f6870cb15531ce3 ea0a2ce364fe417aba3aa8d47a377eae] Unexpected build error:            
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume Traceback (most recent call last):                                                                                                              
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it                                          
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     result = runner(context, *args, **kwargs)                                                                                                   
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__                                                       
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     self.result = self.task(*args, **kwargs)                                                                                                    
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__                                 
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     **volume_spec)                                                                                                                              
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source_volume               
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)                                                                     
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 179, in create_cloned_volume                                 
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     self.create_snapshot(temp_snapshot)                                                                                                         
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 446, in create_snapshot                                      
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     self._create_snapshot(snapshot, path_to_disk, snap_id)                                                                                      
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 496, in _create_snapshot                                     
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     self._write_info_file(info_path, snap_info)                                                                                                 
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 521, in _write_info_file                                     
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume     with open(info_path, 'w') as f:                                                                                                             
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume IOError: [Errno 13] Permission denied: u'/var/lib/cinder/mnt/1f79a07997c5c9cea4fdf7d4f0fb7a9f/volume-d0d63bf8-8636-4cae-8dba-f076d756a8c7.info' 
2013-11-13 17:05:27.032 1314 TRACE cinder.volume.flows.create_volume                                                                                                                                                 
2013-11-13 17:05:27.035 1314 ERROR cinder.openstack.common.rpc.amqp [req-11ba4c67-ac02-430b-85bf-4dfdf0694ca4 81b1ae91395f4f408f6870cb15531ce3 ea0a2ce364fe417aba3aa8d47a377eae] Exception during message handling   
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):                                                                                                               
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data                                        
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     **args)                                                                                                                                      
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch                                       
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)                                                                                             
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper                                                                  
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)                                                                                                           
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume                                                   
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     flow.run(context.elevated())                                                                                                                 
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper                                                    
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     return f(self, *args, **kwargs)                                                                                                              
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run                                              
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     run_it(r)                                                                                                                                    
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it                                           
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     self.rollback(context, cause)                                                                                                                
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__                                                                                
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()                                                                                                                              
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it                                           
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     result = runner(context, *args, **kwargs)                                                                                                    
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__                                                        
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     self.result = self.task(*args, **kwargs)
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     **volume_spec)
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source_volume
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 179, in create_cloned_volume
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     self.create_snapshot(temp_snapshot)
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 446, in create_snapshot
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     self._create_snapshot(snapshot, path_to_disk, snap_id)
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 496, in _create_snapshot
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     self._write_info_file(info_path, snap_info)
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 521, in _write_info_file
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp     with open(info_path, 'w') as f:
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp IOError: [Errno 13] Permission denied: u'/var/lib/cinder/mnt/1f79a07997c5c9cea4fdf7d4f0fb7a9f/volume-d0d63bf8-8636-4cae-8dba-f076d756a8c7.info'
2013-11-13 17:05:27.035 1314 TRACE cinder.openstack.common.rpc.amqp

Version-Release number of selected component (if applicable):
OS is: Red Hat Enterprise Linux Server release 6.5 Beta (Santiago)

the Openstack: 
openstack-nova-common-2013.2-4.el6ost.noarch
openstack-nova-console-2013.2-4.el6ost.noarch
openstack-nova-compute-2013.2-4.el6ost.noarch
openstack-nova-conductor-2013.2-4.el6ost.noarch
openstack-nova-scheduler-2013.2-4.el6ost.noarch
openstack-ceilometer-collector-2013.2-1.el6ost.noarch
python-django-openstack-auth-1.1.2-1.el6ost.noarch
openstack-selinux-0.1.2-12.el6ost.noarch
openstack-glance-2013.2-1.el6ost.noarch
openstack-nova-api-2013.2-4.el6ost.noarch
openstack-ceilometer-common-2013.2-1.el6ost.noarch
openstack-nova-network-2013.2-4.el6ost.noarch
openstack-nova-cert-2013.2-4.el6ost.noarch
openstack-dashboard-theme-2013.2-3.el6ost.noarch
redhat-access-plugin-openstack-4.0.0-0.el6ost.noarch
openstack-keystone-2013.2-1.el6ost.noarch
openstack-utils-2013.2-1.el6ost.noarch
openstack-ceilometer-compute-2013.2-1.el6ost.noarch
openstack-nova-novncproxy-2013.2-4.el6ost.noarch
openstack-ceilometer-central-2013.2-1.el6ost.noarch
openstack-ceilometer-api-2013.2-1.el6ost.noarch
openstack-dashboard-2013.2-3.el6ost.noarch
openstack-packstack-2013.2.1-0.9.dev840.el6ost.noarch
openstack-cinder-2013.2-2.el6ost.noarch

How reproducible:
everytime

Steps to Reproduce:
1. create a volume 
2. create another volume using source-volid
3.

Actual results:
error 

Expected results:
the new volume is available

Additional info:

Comment 2 Eric Harney 2013-11-13 20:33:40 UTC

What permissions do /var/lib/cinder/mnt/1f79a07997c5c9cea4fdf7d4f0fb7a9f/volume-d0d63bf8-8636-4cae-8dba-f076d756a8c7.info and /var/lib/cinder/mnt/1f79a07997c5c9cea4fdf7d4f0fb7a9f/ have?

Comment 3 Yogev Rabl 2013-11-14 07:37:30 UTC

(In reply to Eric Harney from comment #2)
> What permissions do
> /var/lib/cinder/mnt/1f79a07997c5c9cea4fdf7d4f0fb7a9f/volume-d0d63bf8-8636-
> 4cae-8dba-f076d756a8c7.info and
> /var/lib/cinder/mnt/1f79a07997c5c9cea4fdf7d4f0fb7a9f/ have?

here are the permissions:

drwxr-xr-x 3 root root 4096 Nov 13 17:05 1f79a07997c5c9cea4fdf7d4f0fb7a9f

and sub-directories:

-rw-r--r-- 1 root root     197120 Nov 13 16:59 volume-4b08a7b4-b2e3-4f9f-92b1-d88592f5633b.tmp-snap-4b08a7b4-b2e3-4f9f-92b1-d88592f5633b
-rw-rw-rw- 1 root root 8589934592 Nov 13 16:07 volume-cf2d325d-3133-4dcb-99d2-10938cb08d12
-rw-r--r-- 1 root root     197120 Nov 13 17:03 volume-cf2d325d-3133-4dcb-99d2-10938cb08d12.tmp-snap-cf2d325d-3133-4dcb-99d2-10938cb08d12
-rw-rw-rw- 1 root root 2147483648 Nov 13 15:09 volume-d0d63bf8-8636-4cae-8dba-f076d756a8c7
-rw-r--r-- 1 root root     197120 Nov 13 17:05 volume-d0d63bf8-8636-4cae-8dba-f076d756a8c7.tmp-snap-d0d63bf8-8636-4cae-8dba-f076d756a8c7

Comment 4 Eric Harney 2013-11-20 14:54:24 UTC

This issue is being worked on in bug 1016798, which will handle GlusterFS share permissions issues related to snapshots.  (The failure here is in creating the temporary snapshot used for cloning.)

*** This bug has been marked as a duplicate of bug 1016798 ***