Bug 1452704 - [RFE] Backport Cinder Multibackend availability support
Summary: [RFE] Backport Cinder Multibackend availability support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 11.0 (Ocata)
Hardware: All
OS: Linux
high
high
Target Milestone: z4
: 11.0 (Ocata)
Assignee: Gorka Eguileor
QA Contact: Tzach Shefi
URL:
Whiteboard:
Depends On: 1466000
Blocks: 1488390 1510931
TreeView+ depends on / blocked
 
Reported: 2017-05-19 13:51 UTC by Luca Miccini
Modified: 2019-02-17 03:26 UTC (History)
11 users (show)

Fixed In Version: openstack-cinder-10.0.6-5.el7ost
Doc Type: Enhancement
Doc Text:
Adds support for multiple Availability Zones within a single Block Storage volume service by defining the Availability Zones on each driver section.
Clone Of:
: 1488390 1510931 (view as bug list)
Environment:
Last Closed: 2018-02-13 16:29:16 UTC
Target Upstream Version:
scohen: needinfo+


Attachments (Terms of Use)
test patch (8.03 KB, patch)
2017-05-29 06:12 UTC, Luca Miccini
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 433437 'None' MERGED Add per-backend availability zones 2020-04-09 13:39:37 UTC
OpenStack gerrit 467115 'None' ABANDONED Add per-backend availability zones 2020-04-09 13:39:37 UTC
Red Hat Product Errata RHBA-2018:0306 normal SHIPPED_LIVE openstack-cinder bug fix advisory 2018-02-14 00:16:06 UTC

Description Luca Miccini 2017-05-19 13:51:35 UTC
Description of problem:

I would like to request the backport of the following cinder feature:

https://blueprints.launchpad.net/cinder/+spec/multibackend-az-support

https://review.openstack.org/#/c/433437/

"Currently, all Cinder services on one node must be part of the same storage availability zone due to this option being in the DEFAULT config section. This made sense when only one backend was specified. But since adding multibackend support, and deprecating support for defining backends in the DEFAULT section, this puts an unnecessary restriction on backend service setup.

The API and scheduler service already need to be made available outside of a given availability zone/fault domain. Since the majority of backends are external devices, making the c-vol service just a control path like the api and scheduler service, it should be fine to include c-vol in with these others. The backends themselves can be in different fault domains. We should therefore allow defining per-backend availability zones."


Version-Release number of selected component (if applicable):

openstack-cinder-10 (ocata)

How reproducible:

always

Steps to Reproduce:
1. verify no backend_availability_zone option is available in cinder.conf
2.
3.

Actual results:

1. AZ value cannot be defined on a per-backend basis
2. cinder-volume service can only manage storage backend in the same (single) AZ defined in cinder.conf

Expected results:

1. cinder to allow the definition of AZ value on a per backend basis.
2. single cinder-volume instance to manage multiple backends in multiple AZ's

Additional info:

Comment 4 Luca Miccini 2017-05-23 10:29:28 UTC
proposed backport: https://review.openstack.org/#/c/467115/

Comment 6 Luca Miccini 2017-05-29 06:12:59 UTC
Created attachment 1283164 [details]
test patch

Comment 14 Tzach Shefi 2018-02-05 09:27:38 UTC
Gorka, let me know if verification below are sufficient to close this(cloned #3). 

Tested on openstack-cinder-10.0.6-9.el7ost.noarch


1. On a system with ceph backend, added second nfs backend. 
2. Added backend_availability_zone dc1 dc2 one per backend.

[tripleo_ceph]
volume_backend_name=tripleo_ceph
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=qe
rbd_pool=qe-volumes
rbd_secret_uuid=cdf24729-bd06-46c2-ba47-7018ae220197
backend_host=hostgroup
backend_availability_zone = dc1

[nfs]
volume_backend_name=nfs
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/nfs_shares.conf
nfs_snapshot_support=True
nas_secure_file_operations=False
nas_secure_file_permissions=False
backend_availability_zone = dc2


[stack@undercloud-0 ~]$ cinder extra-specs-list
+--------------------------------------+------+-----------------------------------------+                                                                                        
| ID                                   | Name | extra_specs                             |                                                                                        
+--------------------------------------+------+-----------------------------------------+                                                                                        
| 178b03de-6c4f-4494-9df5-33cee2927fa1 | ceph | {'volume_backend_name': 'tripleo_ceph'} |                                                                                        
| 99fb86be-2d22-48de-8cd9-332b26204335 | nfs  | {'volume_backend_name': 'nfs'}          |                                                                                        
+--------------------------------------+------+-----------------------------------------+


I'd created two volumes on both backend before adding backend_availability_zone setting, just to make sure backends are OK. 
#cinder create --display-name nfs1 --volume-type nfs 1
#cinder create --display-name ceph1 --volume-type ceph 1

Followed by two more volumes, after backend_availability_zone
#cinder create --display-name nfs2-dc2 --volume-type nfs 1 --availability-zone dc2
#cinder create --display-name ceph1-dc1 --volume-type ceph 1 --availability-zone dc1

Cinder list shows all of them available
[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+                                                                   
| ID                                   | Status    | Name      | Size | Volume Type | Bootable | Attached to |                                                                   
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+                                                                   
| 067d4b76-ac44-40a7-ab44-7a1be7307d2d | available | ceph1-dc1 | 1    | ceph        | false    |             |                                                                   
| 5f58935a-940e-467b-ac2f-5e079b179355 | available | nfs2-dc2  | 1    | nfs         | false    |             |                                                                   
| ddb65abf-a7d4-4422-b25d-7afaa97d9d35 | available | ceph1     | 1    | ceph        | false    |             |                                                                   
| fad1ce35-89c9-46fb-9b76-8d533c57923a | available | nfs1      | 1    | nfs         | false    |             |                                                                   
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+ 

When adding wrong the wrong  --availability-zone, or omitting it altogether (defaults to none) I get an error status as expected. 

#cinder create --display-name ceph1-dc2-fail --volume-type ceph 1 --availability-zone dc2

#cinder create --display-name ceph1-dc2-fail1 --volume-type ceph 1 


| a4109b56-ba39-46c7-9306-f59023caf804 | error     | ceph1-dc2-fail | 1    | ceph        | false    |             |

| 025a5bf7-3d1c-4599-84f8-f5e224c569d9 | error     | ceph1-dc2-fail1 | 1    | ceph        | false    |             |


Anything else I should check before verification?

Comment 15 Tzach Shefi 2018-02-05 09:30:54 UTC
Also cinder show volume, show's correct zone per each volume

cinder show ceph1-dc1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | dc1     

cinder show nfs2-dc2 
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | dc2

Comment 16 Tzach Shefi 2018-02-05 09:36:42 UTC
Also adding cinder service list output

[stack@undercloud-0 ~]$ cinder service-list
+------------------+------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                   | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | hostgroup              | nova | enabled | up    | 2018-02-05T09:36:01.000000 | -               |
| cinder-scheduler | hostgroup              | nova | enabled | up    | 2018-02-05T09:35:57.000000 | -               |
| cinder-volume    | hostgroup@nfs          | dc2  | enabled | up    | 2018-02-05T09:35:52.000000 | -               |
| cinder-volume    | hostgroup@tripleo_ceph | dc1  | enabled | up    | 2018-02-05T09:36:01.000000 | -               |
+------------------+------------------------+------+---------+-------+----------------------------+-----------------+

Comment 17 Gorka Eguileor 2018-02-05 12:11:43 UTC
Verification looks good to me, thanks.

Comment 18 Tzach Shefi 2018-02-05 13:38:50 UTC
Verified on:
openstack-cinder-10.0.6-9.el7ost.noarch
Comments 14-16.

Comment 21 errata-xmlrpc 2018-02-13 16:29:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0306


Note You need to log in before you can comment on or make changes to this bug.