Description of problem: This bug is to track the development of a new driver for Cinder to directly support Red Hat Storage/Gluster being used as a backend for Cinder volumes
Some design notes from thread with eharney: Initial Design: --------------- Initial implementation done by cloning the code that supports NFS, basically doing s/NFS/GlusterFS/, and made the corresponding Nova change for a libvirt driver to run "mount -t glusterfs". This approach was reviewed with the Gluster team and was found acceptable. The initial implementation will focus on using FUSE support for Gluster, since the qemu support for Gluster is still very new and we want something architecturally very simple to get into Grizzly-3. We should be able to quickly follow up with an update in the early Havana development cycle to include support for qemu-kvm native support, but probably we will keep around the FUSE support as well for those users that don't have the qemu integration (other hypervisors perhaps?) RHOS 2.1 will be based on RHEL 6.4 and 6.4 does not yet have the qemu/Gluster integration. This will come in RHEL 6.5, and when we can leverage that we can backport the newer driver with qemu/Gluster support Limitations for the Gluster/FUSE based Driver: ---------------------------------------------- The main thing to note is that it has certain limitations: namely, no volume snapshot or clone support. (Same as the NFS driver.) As far as I can tell this is going to remain the case with this model since there isn't a way to get a consistent snapshot view of the volume data. Questions: ---------- 1) Does this design assume that a prepared/started volume already exists? - I think the answer to that is yes, which I believe is consistent with the rest of the Cinder drivers 2) How does this differ from: http://www.gluster.org/wp-content/uploads/2011/07/Gluster-Openstack-VM-storage-v1-shehjar.pdf - That document describes how to use Gluster as the backing store for the Nova instances directory. Very useful, but separate from Volumes support 3) Should the mount -t glusterfs be done against the virtual IP of the gluster cluster? - Probably. Need Gluster folks to chime in on this 4) Do you verify you have the gluster client code on the hosts? I assume we'll need to bring it as a dependency of nova? - Probably not a Nova dependency because we don't want to bring in clients if the user has decided not to use Gluster support. Either a soft-requires (with a useful error message for when the client is not present) or we make this part of the puppet modules so that if you have Cinder configured to use Gluster, it pulls the gluster client into the Compute Nodes.
(In reply to comment #3) > > 3) Should the mount -t glusterfs be done against the virtual IP of the > gluster cluster? > - Probably. Need Gluster folks to chime in on this The server IP provided with mount -t glusterfs is primarily used for fetching a configuration file, referred commonly as volfile in gluster parlance. Once the volfile is available to the native client, then it goes about communicating with the servers (bricks) directly. Hence additional VIP and failover configuration is usually not necessary with native client. To circumvent the scenario of the mount server being unreachable at the time of mounting, mount.glusterfs provides options like 'backupvolfile-server' and 'fetch-attempts' which can be used to specify additional mount servers and retries respectively.
To use, set "volume_driver = cinder.volume.glusterfs.GlusterfsDriver" in cinder.conf. Also must set "glusterfs_shares_config".
[root@new32 cinder(keystone_admin)]$ cinder create 1 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | created_at | 2013-03-07T19:02:34.938713 | | display_description | None | | display_name | None | | id | e46eb828-d108-4f52-9e8d-269b48340f0b | | metadata | {} | | size | 1 | | snapshot_id | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@new32 cinder(keystone_admin)]$ cinder show e46eb828-d108-4f52-9e8d-269b48340f0b +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | created_at | 2013-03-07T19:02:34.000000 | | display_description | None | | display_name | None | | id | e46eb828-d108-4f52-9e8d-269b48340f0b | | metadata | {} | | size | 1 | | snapshot_id | None | | status | available | | volume_type | None | +---------------------+--------------------------------------+ [root@new32 cinder(keystone_admin)]$ cinder snapshot-create e46eb828-d108-4f52-9e8d-269b48340f0b +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | created_at | 2013-03-07T19:03:19.145674 | | display_description | None | | display_name | None | | id | bab83adc-dbbf-4ce5-8f6d-78e2bf66edab | | size | 1 | | status | creating | | volume_id | e46eb828-d108-4f52-9e8d-269b48340f0b | +---------------------+--------------------------------------+ [root@new32 cinder(keystone_admin)]$ cinder snapshot-show bab83adc-dbbf-4ce5-8f6d-78e2bf66edab +--------------------------------------------+--------------------------------------+ | Property | Value | +--------------------------------------------+--------------------------------------+ | created_at | 2013-03-07T19:03:19.000000 | | display_description | None | | display_name | None | | id | bab83adc-dbbf-4ce5-8f6d-78e2bf66edab | | os-extended-snapshot-attributes:progress | 0% | | os-extended-snapshot-attributes:project_id | 89a38fe6d3194864995ab0872905a65e | | size | 1 | | status | error | | volume_id | e46eb828-d108-4f52-9e8d-269b48340f0b | +--------------------------------------------+--------------------------------------+ [root@new32 cinder(keystone_admin)]$ tail -n 30 volume.log 2013-03-07 19:03:20 5189 TRACE cinder.volume.driver Stderr: ' Volume group "cinder-volumes" not found\n' 2013-03-07 19:03:20 5189 TRACE cinder.volume.driver 2013-03-07 19:03:24 DEBUG cinder.utils [req-d3ee211b-c05b-4200-a77e-729a927ceff1 1d2b871844434bbc99845ed11c3f897b 89a38fe6d3194864995ab0872905a65e] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -L 1G --name _snapshot-bab83adc-dbbf-4ce5-8f6d-78e2bf66edab --snapshot cinder-volumes/volume-e46eb828-d108-4f52-9e8d-269b48340f0b execute /usr/lib/python2.6/site-packages/cinder/utils.py:167 2013-03-07 19:03:24 DEBUG cinder.utils [req-d3ee211b-c05b-4200-a77e-729a927ceff1 1d2b871844434bbc99845ed11c3f897b 89a38fe6d3194864995ab0872905a65e] Result was 5 execute /usr/lib/python2.6/site-packages/cinder/utils.py:184 2013-03-07 19:03:24 5189 ERROR cinder.openstack.common.rpc.amqp [-] Exception during message handling 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 244, in create_snapshot 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp {'status': 'error'}) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp self.gen.next() 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 235, in create_snapshot 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_snapshot(snapshot_ref) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/driver.py", line 211, in create_snapshot 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp '--snapshot', orig_lv_name, run_as_root=True) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/driver.py", line 98, in _try_execute 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp self._execute(*command, **kwargs) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 191, in execute 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp cmd=' '.join(cmd)) 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command. 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -L 1G --name _snapshot-bab83adc-dbbf-4ce5-8f6d-78e2bf66edab --snapshot cinder-volumes/volume-e46eb828-d108-4f52-9e8d-269b48340f0b 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp Exit code: 5 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp Stdout: '' 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp Stderr: ' Volume group "cinder-volumes" not found\n' 2013-03-07 19:03:24 5189 TRACE cinder.openstack.common.rpc.amqp 2013-03-07 19:03:46 5189 DEBUG cinder.manager [-] Running periodic task VolumeManager._publish_service_capabilities periodic_tasks /usr/lib/python2.6/site-packages/cinder/manager.py:164 2013-03-07 19:03:46 5189 DEBUG cinder.manager [-] Running periodic task VolumeManager._report_driver_status periodic_tasks /usr/lib/python2.6/site-packages/cinder/manager.py:164 [root@new32 cinder(keystone_admin)]$ The VG does not exists, the volume created on glusterfs. It attempts to create-snapshot on VG. Even if it is not an implemented operation it must not attempt to do an LVM operation. If it is a not implemented capability, it should say it earlier on the snapshot creation attempt. Note: selinux support should be added as well.
This issue is also present in the NFS driver both in RHOS and upstream stable/folsom, and is exposed in the GlusterFS driver because it made the same assumptions as the NFS driver. In Folsom, these drivers inherit from VolumeDriver which implements snapshot operations using LVM -- they should implement their own NotImplementedError() overrides for these. (This should maybe be split into a different bug? Not sure.)
(In reply to comment #9) > This issue is also present in the NFS driver both in RHOS and upstream > stable/folsom, and is exposed in the GlusterFS driver because it made the > same assumptions as the NFS driver. > Posted at https://review.openstack.org/#/c/23967/ for upstream stable/folsom.
verified against openstack-cinder-2012.2.3-6.el6ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0658.html
Adding a link to the upstream how-to doc, for manually integrating Gluster into Cinder: http://www.gluster.org/community/documentation/index.php/GlusterFS_Cinder (for anyone else getting to this BZ via Google search :))