Bug 1297070 - rhel-osp-director: 8.0 - cinder volumes are created on ceph if the overcloud was deployed with "--ceph-storage-scale 1 --block-storage-scale 1".
rhel-osp-director: 8.0 - cinder volumes are created on ceph if the overcloud ...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 10.0 (Newton)
Assigned To: leseb
Yogev Rabl
: AutomationBlocker, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-08 17:44 EST by Alexander Chuzhoy
Modified: 2016-12-16 11:50 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-16 11:50:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Chuzhoy 2016-01-08 17:44:20 EST
rhel-osp-director: 8.0 - cinder volumes are created on ceph if the overcloud was deployed with "--ceph-storage-scale 1 --block-storage-scale 1". 


Environment:
openstack-tripleo-heat-templates-0.8.7-2.el7ost.noarch
openstack-tripleo-image-elements-0.9.7-1.el7ost.noarch
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-tripleo-common-0.0.2-4.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.2-1.el7ost.noarch
instack-undercloud-2.2.0-1.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud with --ceph-storage-scale 1 --block-storage-scale 1
2. Create cinder volume

Result:
The volume is created on the ceph node.

Expected result:
The cinder volume should be created on the block-storage node.
Comment 3 wes hayutin 2016-02-03 11:18:14 EST
ack.. recreated the bug in CI..
Comment 4 Mike Burns 2016-04-07 17:03:37 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 7 Giulio Fidente 2016-10-12 11:32:17 EDT
When you have multiple backends in Cinder, new volumes will go on the one with the most available free space unless the admin creates different volume types in Cinder, and picks one when creating the volume. It does not seem to be a bug.
Comment 8 seb 2016-10-13 10:39:01 EDT
What's the value in your cinder.conf for 'default_volume_type'?
Comment 9 seb 2016-10-19 12:28:26 EDT
It looks like "default_volume_type" is not exposed in any manifests, so we can take this action item and work on it.
Comment 10 seb 2016-10-19 12:51:50 EDT
There has been a little bit of confusion here, it seems that "default_volume_type" can be set in cinder.conf using ExtraConfig.
With that you can simply set it to your preferred backend and you're good to go. If you create a volume and don't specify any type, the type registered in default_volume_type will be used.

More usage info coming for ExtraConfig by Giulio and then we will close this.
Comment 11 Giulio Fidente 2016-10-19 12:55:23 EDT
For example, assuming a volume type named "tripleo_ceph" is created and it gets mapped to the ceph backend, then to default to such a backend use an additional environment file at deployment time with the following contents:

parameter_defaults:
  ExtraConfig:
    cinder::config::cinder_config:
      DEFAULT/default_volume_type:
        value: tripleo_ceph
Comment 12 Giulio Fidente 2016-10-19 12:58:34 EDT
If no volume types are created, Cinder will by default use the backend with the most available free space.

Alexander, do you think the comments above are useful or shall we try postpone this to 11 and see if we can automatically create volume types for each backend in Cinder and automatically pick a default?
Comment 17 John Fulton 2016-11-08 23:31:16 EST
When I use the following with my 11-4.2 OSP10 puddle:

parameter_defaults:
  ExtraConfig:
    cinder::config::cinder_config:
      DEFAULT/default_volume_type:
        value: tripleo_ceph

The deploy fails and `openstack stack failures list overcloud` shows a duplicate declaration error: 

    Error: Duplicate declaration: Cinder_config[DEFAULT/default_volume_type] is already declared; cannot redeclare at /etc/puppet/modules/cinder/manifests/api.pp:349 on node overcloud-controller-0.localdomain

I think this is because default_volume_type is already declared:

 https://github.com/openstack/puppet-cinder/blob/stable/newton/manifests/api.pp#L344-L349

So rather than try to add it as a new declaration, I tested passing a value for it like this: 

parameter_defaults:
  ExtraConfig:
    cinder::config::cinder_config::default_volume_type: tripleo_ceph

or like this since the original goal in the bug report was to have ceph not be the default. (Note that I also explicitly request both backends and there are other ceph args I'm not including in the paste below) 

parameter_defaults:
  CinderEnableRbdBackend: true  
  CinderEnableIscsiBackend: true
  ExtraConfig:
    cinder::config::cinder_config::default_volume_type: tripleo_iscsi

My results indicate that the new parameter made it to hiera:

[stack@hci-director ~]$ openstack server list
+---------------------+---------------------+--------+---------------------+----------------+
| ID                  | Name                | Status | Networks            | Image Name     |
+---------------------+---------------------+--------+---------------------+----------------+
| 00615513-ba8e-44af- | overcloud-compute-0 | ACTIVE | ctlplane=192.168.1. | overcloud-full |
| 8210-d29b930b6ada   |                     |        | 22                  |                |
| 22eaac95-80bf-47d6  | overcloud-          | ACTIVE | ctlplane=192.168.1. | overcloud-full |
| -b14c-802b9b0a7cd9  | blockstorage-0      |        | 33                  |                |
| 42b163c8-8181-4f1b- | overcloud-          | ACTIVE | ctlplane=192.168.1. | overcloud-full |
| a972-0b2fc0298781   | cephstorage-0       |        | 30                  |                |
| 1e3a7bd8-772a-4c29- | overcloud-          | ACTIVE | ctlplane=192.168.1. | overcloud-full |
| 8f23-b826c481c497   | controller-0        |        | 21                  |                |
+---------------------+---------------------+--------+---------------------+----------------+
[stack@hci-director ~]$ 

[stack@hci-director ~]$ for ip in `echo 192.168.1.{2{1,2},3{0,3}}`; do echo $ip; ssh heat-admin@$ip "sudo hiera cinder::config::cinder_config::default_volume_type"; done 2> /dev/null
192.168.1.21
tripleo_iscsi
192.168.1.22
tripleo_iscsi
192.168.1.30
tripleo_iscsi
192.168.1.33
tripleo_iscsi
[stack@hci-director ~]$ 

However, I do not see it in the cinder.conf: 

[stack@hci-director cinder]$ for ip in `echo 192.168.1.{2{1,2},3{0,3}}`; do echo $ip; ssh heat-admin@$ip "sudo grep default_volume_type /etc/cinder/cinder.conf"; done 2> /dev/null
192.168.1.21
#default_volume_type = <None>
192.168.1.22
#default_volume_type = <None>
192.168.1.30
#default_volume_type = <None>
192.168.1.33
#default_volume_type = <None>
[stack@hci-director cinder]$ 

I think the next question is: why does puppet-cinder not update the cinder.conf with the default_volume_type when that same parameter is passed? 

The results of my testing from a user perspective are consistent with the original report. Both backends are configured and work but if I don't pass --volume_type to `cinder create`, then the default volume is a ceph volume. Note: the user should set the default in Heat and not assume block storage will be the default just because --block-storage-scale was passed as an argument. I think the bug here may be that setting a default_volume_type in Heat should should result in the cinder.conf having that parameter set to the value in Heat. Perhaps looking at puppet-cinder is the next step.
Comment 18 Giulio Fidente 2016-11-09 04:32:12 EST
hi John, you need to use the following:

parameter_defaults:
  ExtraConfig:
    cinder::api::default_volume_type: tripleo_iscsi
Comment 19 John Fulton 2016-11-09 09:29:30 EST
Giulio,

Thanks. I'm glad it was just a mistake on my part (I hadn't read the puppet/hiera translation correctly). I confirm that the above produces the desired results in the cinder.conf of the controller node. I will follow up with complete steps to reach the desired configuration. 

  John

[root@overcloud-controller-0 ~]# cat /etc/cinder/cinder.conf | grep -v \# | egrep -v "^$" | head 
[DEFAULT]
glance_api_servers = http://172.16.1.10:9292
glance_api_version = 2
enable_v3_api = True
host = hostgroup
storage_availability_zone = nova
default_availability_zone = nova
default_volume_type = tripleo_iscsi
enabled_backends = tripleo_iscsi,tripleo_ceph,
osapi_volume_listen = 192.168.2.19
[root@overcloud-controller-0 ~]#
Comment 20 John Fulton 2016-11-09 12:21:19 EST
Below are steps to achieve the configuration originally desired per 
the bug report. However, the supportability of this scenario is 
another matter. 

Scenario: Deploy an OSP10 that satisfies the following:
- node types: controller, compute, block-storage, ceph-storage
- all storage uses ceph by default except cinder
- cinder's default storage type should be block-storage
- cinder also supports ceph-storage if the user requests it
Note: LVM is not a supported backend and should only be used for POCs

Deployment: 

Use the following cinder parameter_defaults:  

parameter_defaults:
  CinderEnableRbdBackend: true
  CinderEnableIscsiBackend: true
  ExtraConfig:
    cinder::api::default_volume_type: tripleo_iscsi

Other options, like CephStorageExtraConfig may be necessary, but the
above points out cinder specific configurations. When deploying pass
'--ceph-storage-scale 1 --block-storage-scale 1' and other options.

Post-deployment:

Both volume types must be created within the tenant because Cinder's 
default_volume_type picks the default type if none is specified, but
the type needs to be created first. The following creates both types:

 source overcloudrc
 cinder type-create tripleo_iscsi
 cinder type-key tripleo_iscsi set volume_backend_name=tripleo_iscsi
 cinder type-create tripleo_ceph
 cinder type-key tripleo_ceph set volume_backend_name=tripleo_ceph

Create a volume and do not specify the type. Observe the default type, tripleo_iscsi is created: 

[stack@hci-director ~]$ cinder create --display_name default_volume 5
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2016-11-09T15:32:03.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 9a29fe3e-f0b5-4f91-8dab-89857c10eddc |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | default_volume                       |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 4ee87574c2e54860a4fb7d2555dd5ea1     |
| replication_status             | disabled                             |
| size                           | 5                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | 0f6c1e734d624cad9454ec02b1f90843     |
| volume_type                    | tripleo_iscsi                        |
+--------------------------------+--------------------------------------+
[stack@hci-director ~]$ cinder list
+--------------------------------------+-----------+----------------+------+---------------+----------+-------------+
| ID                                   | Status    | Name           | Size | Volume Type   | Bootable | Attached to |
+--------------------------------------+-----------+----------------+------+---------------+----------+-------------+
| 9a29fe3e-f0b5-4f91-8dab-89857c10eddc | available | default_volume | 5    | tripleo_iscsi | false    |             |
+--------------------------------------+-----------+----------------+------+---------------+----------+-------------+
[stack@hci-director ~]$ 

If a user then wants to create a cinder volume with a ceph backend, then they need to
pass a --volume_type tripleo_ceph argument like this: 

 cinder create --volume_type tripleo_ceph --display_name ceph_vol 5

Note that two cinder volumes were created of the desired type and that the type only had to be specified for Ceph. 

[stack@hci-director ~]$ cinder list
+--------------------------------------+-----------+----------------+------+---------------+----------+--------------------------------------+
| ID                                   | Status    | Name           | Size | Volume Type   | Bootable | Attached to                          |
+--------------------------------------+-----------+----------------+------+---------------+----------+--------------------------------------+
| 9a29fe3e-f0b5-4f91-8dab-89857c10eddc | in-use    | default_volume | 5    | tripleo_iscsi | false    | bc25a5bd-beca-413e-9b31-969bdbcc945a |
| d0b77ba7-f75a-4625-82c7-229d76874b82 | available | ceph_vol       | 5    | tripleo_ceph  | false    |                                      |
+--------------------------------------+-----------+----------------+------+---------------+----------+--------------------------------------+
[stack@hci-director ~]$

Note You need to log in before you can comment on or make changes to this bug.