Bug 1269990 - [RFE] support for cinder multiple heterogeneous backends for a single deployment
[RFE] support for cinder multiple heterogeneous backends for a single deployment
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
8.0 (Liberty)
All Unspecified
unspecified Severity high
: ga
: 8.0 (Liberty)
Assigned To: Giulio Fidente
lkuchlan
: FutureFeature, OtherQA
: 1280005 1295033 (view as bug list)
Depends On: 1291943 1312000
Blocks: 1261979 1339413 1266220 1302085 1310828 1334519 1339836
  Show dependency treegraph
 
Reported: 2015-10-08 13:50 EDT by arkady kanevsky
Modified: 2016-10-12 23:34 EDT (History)
25 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-0.8.11-1.el7ost
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1312000 1339836 (view as bug list)
Environment:
Last Closed: 2016-04-07 17:41:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2213581 None None None 2016-10-12 23:34 EDT
OpenStack gerrit 269534 None None None 2016-02-18 09:40 EST
OpenStack gerrit 282332 None None None 2016-02-25 08:31 EST
OpenStack gerrit 289979 None None None 2016-03-10 07:41 EST

  None (edit)
Description arkady kanevsky 2015-10-08 13:50:09 EDT
Description of problem:
Heat template for supporting multiple backend in single deployment
Minimum set of backends are: Ceph, Dell PS, Dell SC.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Expected to be on par with current functionality of OFI for JetStream 4.0.1 solution
Comment 6 Mike Orazi 2016-02-17 14:31:56 EST
Giulio/Steve,

I think there has been some discussion for this item on mailing lists, but I'm hoping we can get that information onto the bugzilla.

I _believe_ that the current state of the world is that we can in fact specify multiple cinder backends via tht as long as the cinder backend type is distinct.  

I also _believe_ that there was a suggestion that in a future release ChainedResources would provide a more elegant solution, but in the interim we could workaround the issue using PostConfig hooks.

Can I get confirmation/correction on the bits that I got wrong above?
Comment 8 Giulio Fidente 2016-02-18 05:04:30 EST
Mike, as of today it was possible to define additional arbitrary backends (passing its config stanza via ExtraConfig) but these had to be enabled via a PostDeployment script.

We are working on a patch [1] which will make it possible to pass via ExtraConfig a list of additional backends to be enabled so that we won't need the PostDeployment script. This should also work regardless of the actual backend type.

1. https://review.openstack.org/269534
Comment 9 Giulio Fidente 2016-02-18 09:49:16 EST
*** Bug 1295033 has been marked as a duplicate of this bug. ***
Comment 10 Mike Burns 2016-02-25 08:31:09 EST
split this bug into OPM and THT.
Comment 12 Mike Burns 2016-03-04 09:56:51 EST
*** Bug 1280005 has been marked as a duplicate of this bug. ***
Comment 14 Giulio Fidente 2016-03-09 04:29:50 EST
When the feature will be available, to configure any additional backend one should just need to provide the backend config settings via a heat environment file, like the following:

parameter_defaults:
  ExtraConfig:
    cinder::config::cinder_config:
      mybackend/volume_group:
        value: cinder-volumes
      mybackend/volume_driver:
        value: cinder.volume.drivers.lvm.LVMISCSIDriver
      mybackend/volume_backend_name:
        value: user_facing_backend_name
    cinder_user_enabled_backends: ['mybackend']

In this case, we're adding a new backend named 'mybackend' of type lvm.
Comment 15 arkady kanevsky 2016-03-09 11:51:43 EST
Can you add an example of specification that includes multiple backends?

Do you do multiple env. files?
Do you repeat this block
cinder::config::cinder_config:
      mybackend/volume_group:
        value: cinder-volumes
      mybackend/volume_driver:
        value: cinder.volume.drivers.lvm.LVMISCSIDriver
      mybackend/volume_backend_name:
        value: user_facing_backend_name
    cinder_user_enabled_backends: ['mybackend']
for each backend in the same env. file?
Comment 16 arkady kanevsky 2016-03-09 11:54:17 EST
is there Liberty backport for https://review.openstack.org/#/c/269534/?
Or you cherrypick the fix into RDO manager and OSP director from Mitaka?
Comment 17 Rajini Karthik 2016-03-09 11:58:16 EST
Here is an example of our heat environment file for EQL and Storage Center. As you can see, the resource_registry key doesn't let us add both the templates to the same environment file. Can you tell us how we can use this feature for the example below.


# A Heat environment file which can be used to enable a
# a Cinder  Dell Eqlx backend and Dell Storage Center,
# backend configured via puppet

resource_registry:
OS::TripleO::ControllerExtraConfigPre: /home/osp_admin/pilot/templates/overcloud/puppet/extraconfig/pre_deploy/controller/cinder-eqlx.yaml
OS::TripleO::ControllerExtraConfigPre: /home/osp_admin/pilot/templates/overcloud/puppet/extraconfig/pre_deploy/controller/cinder-dellsc.yaml


parameter_defaults:
#Eqlx
  CinderEnableEqlxBackend: true
  CinderEqlxBackendName: 'tripleo_eqlx'
  CinderEqlxSanIp: '192.168.170.200'
  CinderEqlxSanLogin: 'grpadmin'
  CinderEqlxSanPassword: 'xxx'
  CinderEqlxSanThinProvision: true
  CinderEqlxGroupname: 'group-0'
  CinderEqlxPool: 'default'
  CinderEqlxChapLogin: ''
  CinderEqlxChapPassword: ''
  CinderEqlxUseChap: false

#Dell sc
  CinderEnableDellScBackend: true
  CinderDellScBackendName: 'tripleo_dellsc'
  CinderDellScSanIp: '192.168.190.195'
  CinderDellScSanLogin: 'crowbar'
  CinderDellScSanPassword: 'xxx'
  CinderDellScSsn: '61590'
  CinderDellScIscsiIpAddress: '192.168.170.10'
  CinderDellScIscsiPort: '3260'
  CinderDellScApiPort: '3033'
  CinderDellScServerFolder: 'dellsc_server'
  CinderDellScVolumeFolder: 'dellsc_volume'
Comment 18 Giulio Fidente 2016-03-09 12:19:21 EST
The name of the backend in cinder is what makes it possible to define same type multiple times. Here is what you could do:

parameter_defaults:
  ExtraConfig:
    cinder::config::cinder_config:
      dell1/san_ip:
        value: your_value
      dell1/san_login:
        value: your_value
      dell1/dell_sc_ssn:
        value: your_value
      ... continues with all needed driver options ...
      dell1/volume_driver:
        value: dell.dell_storagecenter_iscsi.DellStorageCenterISCSIDriver
      dell1/volume_backend_name:
        value: dell1
      dell2/san_ip:
        value: your_value
      dell2/san_login:
        value: your_value
      dell2/dell_sc_ssn:
        value: your_value
      ... continues with all needed driver options ...
      dell2/volume_driver:
        value: dell.dell_storagecenter_iscsi.DellStorageCenterISCSIDriver
      dell2/volume_backend_name:
        value: dell2
    cinder_user_enabled_backends: ['dell1','dell2']

This is a bit like pasting section/key:value objects into the cinder.conf file. The section name is indeed arbitrary; the param names as the volume_driver depend on the driver itself.
Comment 19 Giulio Fidente 2016-03-09 12:20:44 EST
Also, this does *not* go overriding any resource_registry resource. You just paste it into a yaml and deply adding to the deploy cmdline:

  -e /path/to/the/backends.yaml
Comment 20 Mike Burns 2016-03-10 07:41:24 EST
(In reply to arkady kanevsky from comment #16)
> is there Liberty backport for https://review.openstack.org/#/c/269534/?
> Or you cherrypick the fix into RDO manager and OSP director from Mitaka?

https://review.openstack.org/289979

This is merged.  It won't make beta 8, but should be in beta 9.

Also, chatted with Giulio offline about this bug a bit and wanted to be sure everything is clear.

The original method that Rajini mentions in comment 17 will still work, but has limitations.  You can have multiple backends, but only a single instance of each backend.

The new method that Giulio introduced here and explained in comment 18 allows for multiple instances of each backend and as many types of supported backends as you want.
Comment 22 arkady kanevsky 2016-03-22 10:33:12 EDT
Mike, 
Which Beta had it landed?
Thanks,
Arkady
Comment 23 Mike Burns 2016-03-22 10:40:13 EDT
(In reply to arkady kanevsky from comment #22)
> Mike, 
> Which Beta had it landed?
> Thanks,
> Arkady

Beta 9
Comment 24 lkuchlan 2016-03-23 10:07:35 EDT
Tested using:
openstack-puppet-modules-7.0.15-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.12-2.el7ost.noarch

Verification flow:

* Updating cinder.conf of overcloud nodes for 2 NetApp back-ends via a heat environment file

[stack@instack ~]$ cat /mulit_backends.yaml 
parameter_defaults:
  ExtraConfig:
    cinder::config::cinder_config:
        netapp1/volume_driver:
                value: cinder.volume.drivers.netapp.common.NetAppDriver
        netapp1/netapp_storage_family:
                value: ontap_7mode
        netapp1/netapp_storage_protocol:
                value: iscsi
        netapp1/netapp_server_hostname:
                value: 10.35.64.11
        netapp1/netapp_server_port:
                value: 80
        netapp1/netapp_login:
                value: root
        netapp1/netapp_password:
                value: 123456
        netapp1/volume_backend_name:
                value: netapp_1
        netapp2/volume_driver:
                value: cinder.volume.drivers.netapp.common.NetAppDriver
        netapp2/netapp_storage_family:
                value: ontap_7mode
        netapp2/netapp_storage_protocol:
                value: iscsi   
        netapp2/netapp_server_hostname:
                value: 10.35.64.11
        netapp2/netapp_server_port:
                value: 80
        netapp2/netapp_login:
                value: root  
        netapp2/netapp_password:
                value: 123456  
        netapp2/volume_backend_name:
                value: netapp_2
    cinder_user_enabled_backends: ['netapp1','netapp2']


* Overcloud deployment

[stack@instack ~]$ openstack overcloud deploy --templates -e /mulit_backends.yaml  --control-scale 1 --compute-scale 1 --debug --log-file=overcloud_deploy.log

* Content of cinder.conf file after overcloud deloyment 

[root@overcloud-controller-0 ~]# cat /etc/cinder/cinder.conf | grep -C 8  'netapp1\|netapp2'

#enabled_backends = <None>
enabled_backends = tripleo_iscsi,netapp1,netapp2


[netapp1]
volume_backend_name=netapp_1
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_login=root
netapp_storage_protocol=iscsi
netapp_password=123456
netapp_storage_family=ontap_7mode
netapp_server_port=80
netapp_server_hostname=10.35.64.11

[netapp2]
volume_backend_name=netapp_2
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_login=root
netapp_storage_protocol=iscsi
netapp_password=123456
netapp_storage_family=ontap_7mode
netapp_server_port=80
netapp_server_hostname=10.35.64.11

* Volume types for Netapp back-ends

[stack@instack ~]$ cinder extra-specs-list
+--------------------------------------+---------+---------------------------------------+
|                  ID                  |   Name  |              extra_specs              |
+--------------------------------------+---------+---------------------------------------+
| 67e641bd-8c44-4c2a-87ff-bcfe1027fa6d | netapp1 | {u'volume_backend_name': u'netapp_1'} |
| 84d7d01d-4a0b-4ccb-9f0e-21df6e37863d | netapp2 | {u'volume_backend_name': u'netapp_2'} |
+--------------------------------------+---------+---------------------------------------+

* Creating a volume on both back-ends

[stack@instack ~]$ cinder list
+--------------------------------------+-----------+------------------+------+------+-------------+----------+-------------+-------------+
|                  ID                  |   Status  | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------------------+------+------+-------------+----------+-------------+-------------+
| 703f9d13-e60a-457f-9010-baf0b3d621cb | available |        -         |  -   |  1   |   netapp1   |  false   |    False    |             |
| bed7cc29-74e3-4bd4-809c-1fd348c6f92d | available |        -         |  -   |  1   |   netapp2   |  false   |    False    |             |
+--------------------------------------+-----------+------------------+------+------+-------------+----------+-------------+-------------+
Comment 25 Rajini Karthik 2016-04-04 14:42:03 EDT
Verified with dell storage backends as well on Beta9
Comment 27 Rajini Karthik 2016-04-07 11:54:14 EDT
This is causing problems when deployed with Ceph

https://bugzilla.redhat.com/show_bug.cgi?id=1324932
Comment 28 Giulio Fidente 2016-04-07 15:29:47 EDT
(In reply to Rajini Ram from comment #27)
> This is causing problems when deployed with Ceph
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1324932

I was able to deploy using user-provisioned backends and ceph successfully. If there is a problem I think it is unrelated to this change.
Comment 29 errata-xmlrpc 2016-04-07 17:41:13 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html

Note You need to log in before you can comment on or make changes to this bug.