1649845 – cinder backup fail while backend is in different availability zone

Bug 1649845 - cinder backup fail while backend is in different availability zone

Summary: cinder backup fail while backend is in different availability zone

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-cinder
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Cinder Bugs List
QA Contact:	Avi Avraham
Docs Contact:	Kim Nylander
URL:
Whiteboard:
Duplicates (1):	1715469 (view as bug list)
Depends On:
Blocks:	1554013 1650290
TreeView+	depends on / blocked

Reported:	2018-11-14 15:56 UTC by Avi Avraham
Modified:	2023-04-05 15:59 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1650290 (view as bug list)
Environment:
Last Closed:	2018-11-22 12:30:08 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-23999	0	None	None	None	2023-04-05 15:59:35 UTC

Description Avi Avraham 2018-11-14 15:56:25 UTC

Description of problem:
While trying to verify RFE 1554013 we faceed the following error 
ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-cf554ab9-e9bd-452a-8790-61d873afd08f)


Version-Release number of selected component (if applicable):
openstack-cinder-13.0.1-0.20181013185427.31ff628.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. setup cinder backup 
2. configure backend in different AZ then backup backend  
3. create volume from image 
4. run the following command : 
openstack --debug volume backup create --name vol-cirros-backup vol-cirros

Actual results:
backup failed with the following error 
ClientException: Service cinder-backup could not be found. (HTTP 503) 

Expected results:
volume will be backed-up 

Additional info:
(overcloud) [stack@undercloud-0 ~]$ openstack --debug volume backup create --name vol-cirros-backup vol-cirros
START with options: [u'--debug', u'volume', u'backup', u'create', u'--name', u'vol-cirros-backup', u'vol-cirros']
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='http://10.0.0.109:5000/v3', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.46', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='2', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='3', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.109:5000/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '3', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'interface': None, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': u'1', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', u'baremetal_status_code_retries': 5, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'username': 'admin', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.109:5000/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', u'clustering_api_version': u'1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'block_storage_api_version': u'2', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '3', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 3, cmd group openstack.volume.v3
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
key_manager API version 1, cmd group openstack.key_manager.v1
orchestration API version 1, cmd group openstack.orchestration.v1
baremetal API version 1.46, cmd group openstack.baremetal.v1
messaging API version 2, cmd group openstack.messaging.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
data_processing API version 1.1, cmd group openstack.data_processing.v1
container_infra API version 1, cmd group openstack.container_infra.v1
alarming API version 2, cmd group openstack.alarming.v2
dns API version 2, cmd group openstack.dns.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
database API version 1, cmd group openstack.database.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.109:5000/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '3', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.109:5000/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '3', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
command: volume backup create -> openstackclient.volume.v2.backup.CreateVolumeBackup (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.109:5000/v3', 'additional_user_agent': [('osc-lib', '1.11.1')], u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '3', 'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://10.0.0.109:5000/v3', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET http://10.0.0.109:5000/v3 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.17.2 keystoneauth1/3.10.0 python-requests/2.19.1 CPython/2.7.5"
Starting new HTTP connection (1): 10.0.0.109
http://10.0.0.109:5000 "GET /v3 HTTP/1.1" 200 192
RESP: [200] Content-Encoding: gzip Content-Length: 192 Content-Type: application/json Date: Wed, 14 Nov 2018 15:43:55 GMT Server: Apache Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-99cdb427-6dbd-460e-a974-e6585c174aab
RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.0.0.109:5000/v3/", "rel": "self"}]}}
GET call to http://10.0.0.109:5000/v3 used request id req-99cdb427-6dbd-460e-a974-e6585c174aab
Making authentication request to http://10.0.0.109:5000/v3/auth/tokens
http://10.0.0.109:5000 "POST /v3/auth/tokens HTTP/1.1" 201 9553
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "17b65d5a1dd04be3957f5551de55ff0a", "name": "reader"}, {"id": "1184837acbb34c949221ca8a6d5ebd18", "name": "member"}, {"id": "bcde172620a8489fa412f38894490a65", "name": "admin"}], "expires_at": "2018-11-14T16:43:56.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "50bc97ff576b4a60b81eca7830eee529", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.17.1.10:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11bac3026b814b0eb4d2577c6b8cf0b1"}, {"url": "http://172.17.1.10:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bf37c4de5dc4476b8f90aa55b81afd50"}, {"url": "http://10.0.0.109:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "e5fd9087d8f840ff94db52c9fcf0204e"}], "type": "compute", "id": "368e5785a42b4263ba41644adc3e577a", "name": "nova"}, {"endpoints": [{"url": "http://10.0.0.109:8776/v2/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "32764bf986df4d4a8de974d6bbc80013"}, {"url": "http://172.17.1.10:8776/v2/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "3cff9181f8684f589ac47d92381cafd9"}, {"url": "http://172.17.1.10:8776/v2/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "7d698ccd854e4c049ea2f0f6d4968130"}], "type": "volumev2", "id": "4d33aaa147d84d92ac33b0d4d5ecea46", "name": "cinderv2"}, {"endpoints": [{"url": "http://10.0.0.109:9311", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "57be403d54a340f39ef6ae8ba46db8a5"}, {"url": "http://172.17.1.10:9311", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "e84cc0f92da747e78d7bfcf0ca74aa32"}, {"url": "http://172.17.1.10:9311", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "fd9f06fc4cd34d8a8259f606211d3bb3"}], "type": "key-manager", "id": "519426854b3b450dbf4125f6691a94e0", "name": "barbican"}, {"endpoints": [{"url": "http://172.17.1.10:8776/v3/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2cd8096b0a8e4e3cbaeaa579f4e69596"}, {"url": "http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "50af256205a14d5596cce0eff43540eb"}, {"url": "http://172.17.1.10:8776/v3/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ef2181f87382408882ed36780f97bacb"}], "type": "volumev3", "id": "5d951d6bbaf7497e929b31aef328415e", "name": "cinderv3"}, {"endpoints": [{"url": "http://10.0.0.109:8004/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "55ba102fc082451bb4667780b7e763e9"}, {"url": "http://172.17.1.10:8004/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "90d5b53368c4450183ad68342b1a6b13"}, {"url": "http://172.17.1.10:8004/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "9da3f2e3dd754249959ce56fe472f05e"}], "type": "orchestration", "id": "8d94296e3d814423ad779657356c1076", "name": "heat"}, {"endpoints": [{"url": "http://172.17.1.10:8977", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "72ed3645c03940dd8d7e08175f6ceff3"}, {"url": "http://172.17.1.10:8977", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a696095c7d334ed2b830883b0731cef7"}, {"url": "http://10.0.0.109:8977", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef84549b545a4cee8220cc3cf7fe53ef"}], "type": "event", "id": "90238e30b3cf4b049046b166c812f13e", "name": "panko"}, {"endpoints": [{"url": "http://10.0.0.109:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "4d3f269667764e7095ef2072a2e4f534"}, {"url": "http://172.17.1.10:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59ac9679a98144d18541ea83920152a5"}, {"url": "http://172.17.1.10:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba0b4307698b4987abdd1a1781b32ffc"}], "type": "network", "id": "954dce9f869e40d49e660f5d8cfd32c6", "name": "neutron"}, {"endpoints": [{"url": "http://10.0.0.109:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "1ac0c86d274942dbaedceb4358e801b6"}, {"url": "http://172.17.1.10:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "970a0177c04e4ae9969c9370b0a93513"}, {"url": "http://172.17.1.10:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "975b5ae8564b4463a08700aa675a002f"}], "type": "placement", "id": "ac792e1465484556be7b25bab5a2b5a0", "name": "placement"}, {"endpoints": [{"url": "http://172.17.3.23:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "6495e455594b4404b063f25c2c4cf23e"}, {"url": "http://10.0.0.109:8080/v1/AUTH_50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "704bce5c8b774c37932d6f8835efccba"}, {"url": "http://172.17.3.23:8080/v1/AUTH_50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "fbb72f55205e4a58af3e56ac3cbe0614"}], "type": "object-store", "id": "b6b5f40dccc841ac97838f2da2316562", "name": "swift"}, {"endpoints": [{"url": "http://172.17.1.10:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "43c49c35ff44421cb75ca0ba2c02b140"}, {"url": "http://172.17.1.10:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "72daf7be5b5540c2a117522019db4da0"}, {"url": "http://10.0.0.109:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "bd4391554a6947318d0174dd4ff7589d"}], "type": "image", "id": "b9da6d98be264a42a7190fd977881fe9", "name": "glance"}, {"endpoints": [{"url": "http://172.17.1.10:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2a79e905e00444fcbd5f4d8fe34ef92b"}, {"url": "http://10.0.0.109:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "e5caae0162734445aca43a25972bdb3c"}, {"url": "http://192.168.24.9:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "fef37e3ff59f4158882cf9b3b726c055"}], "type": "identity", "id": "bb5089ee9a294cd2b27767d13fe6cfee", "name": "keystone"}, {"endpoints": [{"url": "http://10.0.0.109:8042", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "972556c392af44358a59116654fe327e"}, {"url": "http://172.17.1.10:8042", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "cabea869f54a4a858c74745d9ab40fe9"}, {"url": "http://172.17.1.10:8042", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d203dccbd34f48348a0e6bd43c4b3388"}], "type": "alarming", "id": "c9855879cbb0404d8b23215576f5be0d", "name": "aodh"}, {"endpoints": [{"url": "http://172.17.1.10:8000/v1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "39cef9124b48414c9bddeddd685d041e"}, {"url": "http://172.17.1.10:8000/v1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "872154f1a64a40f58516ee8a81588e1a"}, {"url": "http://10.0.0.109:8000/v1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "adab07239b934a8f888f62404799f5b3"}], "type": "cloudformation", "id": "ce4d74b07f4c43289558a07ab3048979", "name": "heat-cfn"}, {"endpoints": [{"url": "http://10.0.0.109:8041", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a75cb1ae46524243a63e4e3e6c9ad4b9"}, {"url": "http://172.17.1.10:8041", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "b5c60d9666b04885b357677d9bbfb1bf"}, {"url": "http://172.17.1.10:8041", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d52a5ee4d1214f2ab7899b58440f133c"}], "type": "metric", "id": "cf6760b8bfe54fc081a09a449391ee0e", "name": "gnocchi"}, {"endpoints": [], "type": "metering", "id": "e750a8b7b7044673b6c25c491bb4fa51", "name": "ceilometer"}, {"endpoints": [{"url": "http://172.17.1.10:8776/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "09027f74225d4566aedcabf3b3e8a67f"}, {"url": "http://172.17.1.10:8776/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0fb530034e4e41eeb8ac25c4fa63a08f"}, {"url": "http://10.0.0.109:8776/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "71f7bed1cc974ced93feaf6023f362eb"}], "type": "volume", "id": "f9550d641d4947fa919ac9304ab40643", "name": "cinder"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "1015400afd2f4c429bb2241282b297c9"}, "audit_ids": ["9ce633DhSpW1wBSX_YK2wg"], "issued_at": "2018-11-14T15:43:56.000000Z"}}
run(Namespace(columns=[], container=None, description=None, fit_width=False, force=False, formatter='table', incremental=False, max_width=0, name=u'vol-cirros-backup', noindent=False, prefix='', print_empty=False, snapshot=None, variables=[], volume=u'vol-cirros'))
Instantiating volume client: <class 'cinderclient.v3.client.Client'>
Making authentication request to http://10.0.0.109:5000/v3/auth/tokens
http://10.0.0.109:5000 "POST /v3/auth/tokens HTTP/1.1" 201 9553
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "17b65d5a1dd04be3957f5551de55ff0a", "name": "reader"}, {"id": "1184837acbb34c949221ca8a6d5ebd18", "name": "member"}, {"id": "bcde172620a8489fa412f38894490a65", "name": "admin"}], "expires_at": "2018-11-14T16:43:57.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "50bc97ff576b4a60b81eca7830eee529", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.17.1.10:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11bac3026b814b0eb4d2577c6b8cf0b1"}, {"url": "http://172.17.1.10:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bf37c4de5dc4476b8f90aa55b81afd50"}, {"url": "http://10.0.0.109:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "e5fd9087d8f840ff94db52c9fcf0204e"}], "type": "compute", "id": "368e5785a42b4263ba41644adc3e577a", "name": "nova"}, {"endpoints": [{"url": "http://10.0.0.109:8776/v2/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "32764bf986df4d4a8de974d6bbc80013"}, {"url": "http://172.17.1.10:8776/v2/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "3cff9181f8684f589ac47d92381cafd9"}, {"url": "http://172.17.1.10:8776/v2/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "7d698ccd854e4c049ea2f0f6d4968130"}], "type": "volumev2", "id": "4d33aaa147d84d92ac33b0d4d5ecea46", "name": "cinderv2"}, {"endpoints": [{"url": "http://10.0.0.109:9311", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "57be403d54a340f39ef6ae8ba46db8a5"}, {"url": "http://172.17.1.10:9311", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "e84cc0f92da747e78d7bfcf0ca74aa32"}, {"url": "http://172.17.1.10:9311", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "fd9f06fc4cd34d8a8259f606211d3bb3"}], "type": "key-manager", "id": "519426854b3b450dbf4125f6691a94e0", "name": "barbican"}, {"endpoints": [{"url": "http://172.17.1.10:8776/v3/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2cd8096b0a8e4e3cbaeaa579f4e69596"}, {"url": "http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "50af256205a14d5596cce0eff43540eb"}, {"url": "http://172.17.1.10:8776/v3/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ef2181f87382408882ed36780f97bacb"}], "type": "volumev3", "id": "5d951d6bbaf7497e929b31aef328415e", "name": "cinderv3"}, {"endpoints": [{"url": "http://10.0.0.109:8004/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "55ba102fc082451bb4667780b7e763e9"}, {"url": "http://172.17.1.10:8004/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "90d5b53368c4450183ad68342b1a6b13"}, {"url": "http://172.17.1.10:8004/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "9da3f2e3dd754249959ce56fe472f05e"}], "type": "orchestration", "id": "8d94296e3d814423ad779657356c1076", "name": "heat"}, {"endpoints": [{"url": "http://172.17.1.10:8977", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "72ed3645c03940dd8d7e08175f6ceff3"}, {"url": "http://172.17.1.10:8977", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a696095c7d334ed2b830883b0731cef7"}, {"url": "http://10.0.0.109:8977", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef84549b545a4cee8220cc3cf7fe53ef"}], "type": "event", "id": "90238e30b3cf4b049046b166c812f13e", "name": "panko"}, {"endpoints": [{"url": "http://10.0.0.109:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "4d3f269667764e7095ef2072a2e4f534"}, {"url": "http://172.17.1.10:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59ac9679a98144d18541ea83920152a5"}, {"url": "http://172.17.1.10:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba0b4307698b4987abdd1a1781b32ffc"}], "type": "network", "id": "954dce9f869e40d49e660f5d8cfd32c6", "name": "neutron"}, {"endpoints": [{"url": "http://10.0.0.109:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "1ac0c86d274942dbaedceb4358e801b6"}, {"url": "http://172.17.1.10:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "970a0177c04e4ae9969c9370b0a93513"}, {"url": "http://172.17.1.10:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "975b5ae8564b4463a08700aa675a002f"}], "type": "placement", "id": "ac792e1465484556be7b25bab5a2b5a0", "name": "placement"}, {"endpoints": [{"url": "http://172.17.3.23:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "6495e455594b4404b063f25c2c4cf23e"}, {"url": "http://10.0.0.109:8080/v1/AUTH_50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "704bce5c8b774c37932d6f8835efccba"}, {"url": "http://172.17.3.23:8080/v1/AUTH_50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "fbb72f55205e4a58af3e56ac3cbe0614"}], "type": "object-store", "id": "b6b5f40dccc841ac97838f2da2316562", "name": "swift"}, {"endpoints": [{"url": "http://172.17.1.10:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "43c49c35ff44421cb75ca0ba2c02b140"}, {"url": "http://172.17.1.10:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "72daf7be5b5540c2a117522019db4da0"}, {"url": "http://10.0.0.109:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "bd4391554a6947318d0174dd4ff7589d"}], "type": "image", "id": "b9da6d98be264a42a7190fd977881fe9", "name": "glance"}, {"endpoints": [{"url": "http://172.17.1.10:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2a79e905e00444fcbd5f4d8fe34ef92b"}, {"url": "http://10.0.0.109:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "e5caae0162734445aca43a25972bdb3c"}, {"url": "http://192.168.24.9:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "fef37e3ff59f4158882cf9b3b726c055"}], "type": "identity", "id": "bb5089ee9a294cd2b27767d13fe6cfee", "name": "keystone"}, {"endpoints": [{"url": "http://10.0.0.109:8042", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "972556c392af44358a59116654fe327e"}, {"url": "http://172.17.1.10:8042", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "cabea869f54a4a858c74745d9ab40fe9"}, {"url": "http://172.17.1.10:8042", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d203dccbd34f48348a0e6bd43c4b3388"}], "type": "alarming", "id": "c9855879cbb0404d8b23215576f5be0d", "name": "aodh"}, {"endpoints": [{"url": "http://172.17.1.10:8000/v1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "39cef9124b48414c9bddeddd685d041e"}, {"url": "http://172.17.1.10:8000/v1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "872154f1a64a40f58516ee8a81588e1a"}, {"url": "http://10.0.0.109:8000/v1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "adab07239b934a8f888f62404799f5b3"}], "type": "cloudformation", "id": "ce4d74b07f4c43289558a07ab3048979", "name": "heat-cfn"}, {"endpoints": [{"url": "http://10.0.0.109:8041", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a75cb1ae46524243a63e4e3e6c9ad4b9"}, {"url": "http://172.17.1.10:8041", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "b5c60d9666b04885b357677d9bbfb1bf"}, {"url": "http://172.17.1.10:8041", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d52a5ee4d1214f2ab7899b58440f133c"}], "type": "metric", "id": "cf6760b8bfe54fc081a09a449391ee0e", "name": "gnocchi"}, {"endpoints": [], "type": "metering", "id": "e750a8b7b7044673b6c25c491bb4fa51", "name": "ceilometer"}, {"endpoints": [{"url": "http://172.17.1.10:8776/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "09027f74225d4566aedcabf3b3e8a67f"}, {"url": "http://172.17.1.10:8776/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0fb530034e4e41eeb8ac25c4fa63a08f"}, {"url": "http://10.0.0.109:8776/v1/50bc97ff576b4a60b81eca7830eee529", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "71f7bed1cc974ced93feaf6023f362eb"}], "type": "volume", "id": "f9550d641d4947fa919ac9304ab40643", "name": "cinder"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "1015400afd2f4c429bb2241282b297c9"}, "audit_ids": ["T7e5NdvJR2-TggjDQ0M8BQ"], "issued_at": "2018-11-14T15:43:57.000000Z"}}
REQ: curl -g -i -X GET http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/volumes/vol-cirros -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}cffe18c0dd5b645a8108306fb083ca32165149ff"
Starting new HTTP connection (1): 10.0.0.109
http://10.0.0.109:8776 "GET /v3/50bc97ff576b4a60b81eca7830eee529/volumes/vol-cirros HTTP/1.1" 404 83
RESP: [404] Content-Length: 83 Content-Type: application/json Date: Wed, 14 Nov 2018 15:43:57 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version x-compute-request-id: req-fa7c28a1-a360-459b-aed4-ded47d35bc55 x-openstack-request-id: req-fa7c28a1-a360-459b-aed4-ded47d35bc55
RESP BODY: {"itemNotFound": {"message": "Volume vol-cirros could not be found.", "code": 404}}
GET call to volumev3 for http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/volumes/vol-cirros used request id req-fa7c28a1-a360-459b-aed4-ded47d35bc55
REQ: curl -g -i -X GET http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/volumes/detail?all_tenants=1&name=vol-cirros -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}cffe18c0dd5b645a8108306fb083ca32165149ff"
http://10.0.0.109:8776 "GET /v3/50bc97ff576b4a60b81eca7830eee529/volumes/detail?all_tenants=1&name=vol-cirros HTTP/1.1" 200 701
RESP: [200] Content-Encoding: gzip Content-Length: 701 Content-Type: application/json Date: Wed, 14 Nov 2018 15:43:58 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version,Accept-Encoding x-compute-request-id: req-0d4c9042-a991-4ab7-a814-ba1120b844dc x-openstack-request-id: req-0d4c9042-a991-4ab7-a814-ba1120b844dc
RESP BODY: {"volumes": [{"migration_status": null, "attachments": [], "links": [{"href": "http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/volumes/d86d9e00-f6ec-43e2-9694-036e2de4c2c2", "rel": "self"}, {"href": "http://10.0.0.109:8776/50bc97ff576b4a60b81eca7830eee529/volumes/d86d9e00-f6ec-43e2-9694-036e2de4c2c2", "rel": "bookmark"}], "availability_zone": "dc2", "os-vol-host-attr:host": "controller-0@nfs#nfs", "encrypted": false, "updated_at": "2018-11-14T15:25:59.000000", "replication_status": null, "snapshot_id": null, "id": "d86d9e00-f6ec-43e2-9694-036e2de4c2c2", "size": 1, "user_id": "1015400afd2f4c429bb2241282b297c9", "os-vol-tenant-attr:tenant_id": "50bc97ff576b4a60b81eca7830eee529", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "volume_image_metadata": {"checksum": "7316af7358dd32ca1956d72ac2c9e147", "min_ram": "0", "disk_format": "qcow2", "image_name": "cirros", "owner_specified.shade.sha256": "53ef7941846012d031db80d2936b23de65c912ae45a1a68b21adb661f080a21b", "image_id": "786bc2f5-d62e-41c6-a979-29b8fd9d9ce0", "signature_verified": "False", "owner_specified.shade.object": "images/cirros", "container_format": "bare", "min_disk": "0", "owner_specified.shade.md5": "7316af7358dd32ca1956d72ac2c9e147", "size": "12528640"}, "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "vol-cirros", "bootable": "true", "created_at": "2018-11-14T15:25:49.000000", "volume_type": "nfs"}]}
GET call to volumev3 for http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/volumes/detail?all_tenants=1&name=vol-cirros used request id req-0d4c9042-a991-4ab7-a814-ba1120b844dc
REQ: curl -g -i -X POST http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/backups -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}cffe18c0dd5b645a8108306fb083ca32165149ff" -H "X-OpenStack-Request-ID: req-0d4c9042-a991-4ab7-a814-ba1120b844dc" -d '{"backup": {"force": false, "description": null, "snapshot_id": null, "incremental": false, "volume_id": "d86d9e00-f6ec-43e2-9694-036e2de4c2c2", "container": null, "name": "vol-cirros-backup"}}'
http://10.0.0.109:8776 "POST /v3/50bc97ff576b4a60b81eca7830eee529/backups HTTP/1.1" 503 93
RESP: [503] Content-Length: 93 Content-Type: application/json Date: Wed, 14 Nov 2018 15:43:59 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version x-compute-request-id: req-cf554ab9-e9bd-452a-8790-61d873afd08f x-openstack-request-id: req-cf554ab9-e9bd-452a-8790-61d873afd08f
RESP BODY: {"serviceUnavailable": {"message": "Service cinder-backup could not be found.", "code": 503}}
POST call to volumev3 for http://10.0.0.109:8776/v3/50bc97ff576b4a60b81eca7830eee529/backups used request id req-cf554ab9-e9bd-452a-8790-61d873afd08f
Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-cf554ab9-e9bd-452a-8790-61d873afd08f)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/openstackclient/volume/v2/backup.py", line 91, in take_action
    snapshot_id=snapshot_id,
  File "/usr/lib/python2.7/site-packages/cinderclient/api_versions.py", line 407, in substitution
    return method.func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 61, in create
    incremental, force, snapshot_id)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 126, in _create_backup
    return self._create('/backups', body, 'backup')
  File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 330, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 203, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
    raise exceptions.from_response(resp, body)
ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-cf554ab9-e9bd-452a-8790-61d873afd08f)
clean_up CreateVolumeBackup: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-cf554ab9-e9bd-452a-8790-61d873afd08f)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/openstackclient/volume/v2/backup.py", line 91, in take_action
    snapshot_id=snapshot_id,
  File "/usr/lib/python2.7/site-packages/cinderclient/api_versions.py", line 407, in substitution
    return method.func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 61, in create
    incremental, force, snapshot_id)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 126, in _create_backup
    return self._create('/backups', body, 'backup')
  File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 330, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 203, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
    raise exceptions.from_response(resp, body)
ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-cf554ab9-e9bd-452a-8790-61d873afd08f)

END return value: 1

Comment 1 Alan Bishop 2018-11-14 16:50:33 UTC

First let's make sure the backup service is up.
- What does "openstack volume service list" report?
- What backup backend are you using?

Comment 2 Avi Avraham 2018-11-15 15:26:26 UTC

+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                        | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | controller-0                | nova | enabled | up    | 2018-11-15T15:25:34.000000 | -               |
| cinder-backup    | controller-1                | nova | enabled | down  | 2018-11-14T15:08:56.000000 | -               |
| cinder-scheduler | controller-0                | nova | enabled | up    | 2018-11-15T15:25:32.000000 | -               |
| cinder-scheduler | controller-1                | nova | enabled | up    | 2018-11-15T15:25:30.000000 | -               |
| cinder-scheduler | controller-2                | nova | enabled | up    | 2018-11-15T15:25:34.000000 | -               |
| cinder-volume    | controller-0@kaminario      | beta | enabled | up    | 2018-11-15T15:25:31.000000 | -               |
| cinder-volume    | controller-0@tripleo_iscsi  | nova | enabled | down  | 2018-11-14T06:43:04.000000 | -               |
| cinder-volume    | controller-0@tripleo_netapp | nova | enabled | down  | 2018-11-13T14:29:11.000000 | -               |
| cinder-volume    | hostgroup@tripleo_ceph      | nova | enabled | down  | 2018-11-14T05:34:05.000000 | -               |
| cinder-volume    | hostgroup@tripleo_netapp    | alfa | enabled | up    | 2018-11-15T15:25:34.000000 | -               |
+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+

Comment 3 Avi Avraham 2018-11-15 15:32:15 UTC

ceph is our backup backend for this setup 
And we have netapp ceph and kaminario as backends
I got a another setup facing the same error with netapp nfs and lvm as backend and swift as backup.

Comment 4 Alan Bishop 2018-11-15 17:38:22 UTC

I verified the feature works, but there are a couple of tricky bits that may
not be documented. The main thing is the feature only works with cinder
API version 3.51 (or later), meaning it requires a microversion.
Unfortunately, due to other factors related to the openstack CLI, TripleO sets
the OS_VOLUME_API_VERSION=3 (no microversion), and that's why the cross-AZ
backup didn't work.

The other thing to know is you have to specify the backup AZ. Compare this
help text:

$ cinder help backup-create
usage: cinder backup-create [--container <container>] [--name <name>]
                            [--description <description>] [--incremental]
                            [--force] [--snapshot-id <snapshot-id>]
                            <volume>

Creates a volume backup.

Positional arguments:
  <volume>              Name or ID of volume to backup.

Optional arguments:
  --container <container>
                        Backup container name. Default=None.
  --name <name>         Backup name. Default=None.
  --description <description>
                        Backup description. Default=None.
  --incremental         Incremental backup. Default=False.
  --force               Allows or disallows backup of a volume when the volume
                        is attached to an instance. If set to True, backs up
                        the volume whether its status is "available" or "in-
                        use". The backup of an "in-use" volume means your data
                        is crash consistent. Default=False.
  --snapshot-id <snapshot-id>
                        ID of snapshot to backup. Default=None.


Now, with the microverion:

$ OS_VOLUME_API_VERSION=3.51 cinder help backup-create
usage: cinder backup-create [--container <container>] [--name <name>]
                            [--description <description>] [--incremental]
                            [--force] [--snapshot-id <snapshot-id>]
                            [--metadata [<key=value> [<key=value> ...]]]
                            [--availability-zone AVAILABILITY_ZONE]
                            <volume>

Creates a volume backup.

Positional arguments:
  <volume>              Name or ID of volume to backup.

Optional arguments:
  --container <container>
                        Backup container name. Default=None.
  --name <name>         Backup name. Default=None.
  --description <description>
                        Backup description. Default=None.
  --incremental         Incremental backup. Default=False.
  --force               Allows or disallows backup of a volume when the volume
                        is attached to an instance. If set to True, backs up
                        the volume whether its status is "available" or "in-
                        use". The backup of an "in-use" volume means your data
                        is crash consistent. Default=False.
  --snapshot-id <snapshot-id>
                        ID of snapshot to backup. Default=None.
  --metadata [<key=value> [<key=value> ...]]
                        Metadata key and value pairs. Default=None. (Supported
                        by API version 3.43 and later)
  --availability-zone AVAILABILITY_ZONE
                        AZ where the backup should be stored, by default it
                        will be the same as the source. (Supported by API
                        version 3.51 and later)

Note the "--availability-zone" parameter. That needs to specify the AZ where
the backup service is running. So, given a volume in AZ "x" and a backup
service running in AZ "y", this command should work:

$ OS_VOLUME_API_VERSION=3.51 cinder backup-create vol --availability-zone y

The final thing to note is the 'openstack' client still does not support
microversions:

$ OS_VOLUME_API_VERSION=3.51 openstack volume backup create ...
volume version 3.51 is not in supported versions: 1, 2, 3

Comment 5 Avi Avraham 2018-11-22 12:29:21 UTC

Alan steps allow us to verify the RFE 
Need to be added to decontamination the required steps.

Comment 6 Tzach Shefi 2019-06-02 08:10:56 UTC

*** Bug 1715469 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.