Bug 1650290

Original engineering BZ: 1554013

+++ This bug was initially created as a clone of Bug #1649845 +++

Description of problem:
While trying to verify RFE 1554013 we faceed the following error 
ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-cf554ab9-e9bd-452a-8790-61d873afd08f)


Version-Release number of selected component (if applicable):
openstack-cinder-13.0.1-0.20181013185427.31ff628.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. setup cinder backup 
2. configure backend in different AZ then backup backend  
3. create volume from image 
4. run the following command : 
openstack --debug volume backup create --name vol-cirros-backup vol-cirros

Actual results:
backup failed with the following error 
ClientException: Service cinder-backup could not be found. (HTTP 503) 

Expected results:
volume will be backed-up 


--- Additional comment from Alan Bishop on 2018-11-14 11:50:33 EST ---

First let's make sure the backup service is up.
- What does "openstack volume service list" report?
- What backup backend are you using?

--- Additional comment from Avi Avraham on 2018-11-15 10:26:26 EST ---

+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                        | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | controller-0                | nova | enabled | up    | 2018-11-15T15:25:34.000000 | -               |
| cinder-backup    | controller-1                | nova | enabled | down  | 2018-11-14T15:08:56.000000 | -               |
| cinder-scheduler | controller-0                | nova | enabled | up    | 2018-11-15T15:25:32.000000 | -               |
| cinder-scheduler | controller-1                | nova | enabled | up    | 2018-11-15T15:25:30.000000 | -               |
| cinder-scheduler | controller-2                | nova | enabled | up    | 2018-11-15T15:25:34.000000 | -               |
| cinder-volume    | controller-0@kaminario      | beta | enabled | up    | 2018-11-15T15:25:31.000000 | -               |
| cinder-volume    | controller-0@tripleo_iscsi  | nova | enabled | down  | 2018-11-14T06:43:04.000000 | -               |
| cinder-volume    | controller-0@tripleo_netapp | nova | enabled | down  | 2018-11-13T14:29:11.000000 | -               |
| cinder-volume    | hostgroup@tripleo_ceph      | nova | enabled | down  | 2018-11-14T05:34:05.000000 | -               |
| cinder-volume    | hostgroup@tripleo_netapp    | alfa | enabled | up    | 2018-11-15T15:25:34.000000 | -               |
+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+

--- Additional comment from Avi Avraham on 2018-11-15 10:32:15 EST ---

ceph is our backup backend for this setup 
And we have netapp ceph and kaminario as backends
I got a another setup facing the same error with netapp nfs and lvm as backend and swift as backup.

--- Additional comment from Alan Bishop on 2018-11-15 12:38:22 EST ---

I verified the feature works, but there are a couple of tricky bits that may
not be documented. The main thing is the feature only works with cinder
API version 3.51 (or later), meaning it requires a microversion.
Unfortunately, due to other factors related to the openstack CLI, TripleO sets
the OS_VOLUME_API_VERSION=3 (no microversion), and that's why the cross-AZ
backup didn't work.

The other thing to know is you have to specify the backup AZ. Compare this
help text:

$ cinder help backup-create
usage: cinder backup-create [--container <container>] [--name <name>]
                            [--description <description>] [--incremental]
                            [--force] [--snapshot-id <snapshot-id>]
                            <volume>

Creates a volume backup.

Positional arguments:
  <volume>              Name or ID of volume to backup.

Optional arguments:
  --container <container>
                        Backup container name. Default=None.
  --name <name>         Backup name. Default=None.
  --description <description>
                        Backup description. Default=None.
  --incremental         Incremental backup. Default=False.
  --force               Allows or disallows backup of a volume when the volume
                        is attached to an instance. If set to True, backs up
                        the volume whether its status is "available" or "in-
                        use". The backup of an "in-use" volume means your data
                        is crash consistent. Default=False.
  --snapshot-id <snapshot-id>
                        ID of snapshot to backup. Default=None.


Now, with the microverion:

$ OS_VOLUME_API_VERSION=3.51 cinder help backup-create
usage: cinder backup-create [--container <container>] [--name <name>]
                            [--description <description>] [--incremental]
                            [--force] [--snapshot-id <snapshot-id>]
                            [--metadata [<key=value> [<key=value> ...]]]
                            [--availability-zone AVAILABILITY_ZONE]
                            <volume>

Creates a volume backup.

Positional arguments:
  <volume>              Name or ID of volume to backup.

Optional arguments:
  --container <container>
                        Backup container name. Default=None.
  --name <name>         Backup name. Default=None.
  --description <description>
                        Backup description. Default=None.
  --incremental         Incremental backup. Default=False.
  --force               Allows or disallows backup of a volume when the volume
                        is attached to an instance. If set to True, backs up
                        the volume whether its status is "available" or "in-
                        use". The backup of an "in-use" volume means your data
                        is crash consistent. Default=False.
  --snapshot-id <snapshot-id>
                        ID of snapshot to backup. Default=None.
  --metadata [<key=value> [<key=value> ...]]
                        Metadata key and value pairs. Default=None. (Supported
                        by API version 3.43 and later)
  --availability-zone AVAILABILITY_ZONE
                        AZ where the backup should be stored, by default it
                        will be the same as the source. (Supported by API
                        version 3.51 and later)

Note the "--availability-zone" parameter. That needs to specify the AZ where
the backup service is running. So, given a volume in AZ "x" and a backup
service running in AZ "y", this command should work:

$ OS_VOLUME_API_VERSION=3.51 cinder backup-create vol --availability-zone y

The final thing to note is the 'openstack' client still does not support
microversions:

$ OS_VOLUME_API_VERSION=3.51 openstack volume backup create ...
volume version 3.51 is not in supported versions: 1, 2, 3