Bug 1715469
| Summary: | Cinder volume backup fails to backup a volume from other AZ | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Tzach Shefi <tshefi> | ||||||
| Component: | openstack-cinder | Assignee: | Cinder Bugs List <cinder-bugs> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Tzach Shefi <tshefi> | ||||||
| Severity: | high | Docs Contact: | Tana <tberry> | ||||||
| Priority: | unspecified | ||||||||
| Version: | 15.0 (Stein) | CC: | abishop | ||||||
| Target Milestone: | --- | Keywords: | Regression | ||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2019-06-02 08:10:56 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Forgot to add full version details puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch openstack-cinder-14.0.1-0.20190507170400.bec06e6.el8ost.noarch python3-cinder-14.0.1-0.20190507170400.bec06e6.el8ost.noarch python3-cinderclient-4.1.0-0.20190312104811.4e17e1d.el8ost.noarch Created attachment 1575215 [details]
Cinder logs from controller-0 and 2
Created attachment 1575223 [details]
Adding full cinder clinet --debug
Full client --debug attached, this is the tail of it.
cinderclient.exceptions.ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-adcc9e42-b81b-4cae-99b7-5ab4db0cf734)
DEBUG:cinderclient.shell:Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-adcc9e42-b81b-4cae-99b7-5ab4db0cf734)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/cinderclient/shell.py", line 905, in main
OpenStackCinderShell().main(sys.argv[1:])
File "/usr/lib/python3.6/site-packages/cinderclient/shell.py", line 728, in main
args.func(self.cs, args)
File "/usr/lib/python3.6/site-packages/cinderclient/v3/shell.py", line 2398, in do_backup_create
**kwargs)
File "/usr/lib/python3.6/site-packages/cinderclient/api_versions.py", line 407, in substitution
return method.func(obj, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/cinderclient/v3/volume_backups.py", line 61, in create
incremental, force, snapshot_id)
File "/usr/lib/python3.6/site-packages/cinderclient/v3/volume_backups.py", line 126, in _create_backup
return self._create('/backups', body, 'backup')
File "/usr/lib/python3.6/site-packages/cinderclient/base.py", line 324, in _create
resp, body = self.api.client.post(url, body=body)
File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 203, in post
return self._cs_request(url, 'POST', **kwargs)
File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 191, in _cs_request
return self.request(url, method, **kwargs)
File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 177, in request
raise exceptions.from_response(resp, body)
cinderclient.exceptions.ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-adcc9e42-b81b-4cae-99b7-5ab4db0cf734)
ERROR: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-adcc9e42-b81b-4cae-99b7-5ab4db0cf734)
Tzach, I'm guessing my comments in bug #1649845 comment #4 apply. Also, see bug #1650993. You probably want to avoid using an NFS backend for testing the backup feature. I'll recheck\update with your tips in mind. cinder --os-volume-api-version 3.51 help backup-create --availability-zone Probably close this as dup or notabug. My cloud was deleted from under me, or rather above (overcloud..) going to retest update this first thing next week. Closing as dup, dup was closed as worksforme. Confirm's Alan's tip, with two separate AZ backed volumes LVM/NFS, backup worked fine once I used -> (overcloud) [stack@undercloud-0 ~]$ cinder --os-volume-api-version 3.51 backup-create ee07078f-3fe3-4f72-957c-37e60b7cfca1 --availability-zone nova --name nfsbackup *** This bug has been marked as a duplicate of bug 1649845 *** |
Description of problem: On a default LVM backed deployment (lvm AZ=nova), I'd manually added a second NFS backend (AZ=dc2). Backup of an LVM volume works fine, however when I try to backup an NFS backed volume I hit a problem backup fails to start, we see client (--debug) error raise exceptions.from_response(resp, body) cinderclient.exceptions.ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-adcc9e42-b81b-4cae-99b7-5ab4db0cf734) ERROR: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-adcc9e42-b81b-4cae-99b7-5ab4db0cf734) Adding background info, there is known issue if backup podman/role moves to other controller we have the original backup in state down. This is the case in my setup, why backup podman/role moved I don't know it was like this before I used the system. (overcloud) [stack@undercloud-0 ~]$ cinder service-list +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | controller-0 | nova | enabled | down | 2019-05-30T08:21:06.000000 | - | | cinder-backup | controller-2 | nova | enabled | up | 2019-05-30T12:38:34.000000 | - | | cinder-scheduler | controller-0 | nova | enabled | up | 2019-05-30T12:38:32.000000 | - | | cinder-scheduler | controller-1 | nova | enabled | up | 2019-05-30T12:38:33.000000 | - | | cinder-scheduler | controller-2 | nova | enabled | up | 2019-05-30T12:38:30.000000 | - | | cinder-volume | controller-0@nfs | dc2 | enabled | up | 2019-05-30T12:38:37.000000 | - | | cinder-volume | hostgroup@tripleo_iscsi | nova | enabled | up | 2019-05-30T12:38:38.000000 | - | +------------------+-------------------------+------+---------+-------+----------------------------+------- Version-Release number of selected component (if applicable): How reproducible: python3-cinderclient-4.1.0-0.20190312104811.4e17e1d.el8ost.noarch puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch Steps to Reproduce: 1. Add second backend in my case nfs with other AZ defined. (overcloud) [stack@undercloud-0 ~]$ cinder availability-zone-list +------+-----------+ | Name | Status | +------+-----------+ | dc2 | available | | nova | available | +------+-----------+ 2. Create two volumes one per backend cinder create --display-name lvm --volume-type tripleo_iscsi 1 --availability-zone nova cinder create --display-name nfs --volume-type nfs 1 --availability-zone dc2 (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+------+------+---------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+---------------+----------+-------------+ | 3c9daaed-6fcb-4fe0-9805-1825890113f3 | available | lvm | 1 | tripleo_iscsi | false | | | 6600c410-c8f8-49ac-9a9b-870de97477aa | available | nfs | 1 | nfs | false | | +--------------------------------------+-----------+------+------+---------------+----------+-------------+ 3. Try to backup both volumes: cinder backup-create 3c9daaed-6fcb-4fe0-9805-1825890113f3 --name lvm Works fine cinder --debug backup-create 6600c410-c8f8-49ac-9a9b-870de97477aa --name nfs -> fails.. with client error (overcloud) [stack@undercloud-0 ~]$ cinder backup-list +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ | 2ac5468c-ccc9-4392-8f77-e7f7497355e0 | 3c9daaed-6fcb-4fe0-9805-1825890113f3 | available | lvm | 1 | 21 | volumebackups | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ Actual results: Cross AZ backup fails for none default NFS/AZ DC2 Expected results: Cross AZ backup was added as an RFE on 14 or 13 backup should work for both volumes Additional info: