We expect cinder-backup (in degraded mode) to work if you follow our docs to deploy storage at central and a dcn site provided that the following is done: 1. "-e environments/cinder-backup.yaml" is used when deploying central [1]. 2. The "openstack" cephx keyring is used at all sites and not "external" How to do the second item above (including doc updates) is tacked by bz 1857776, 1879518. Though we expect it to work product management has requested this TestOnly RFE in order to backups in a DCN context. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/distributed_compute_node_and_storage_deployment/assembly_deploying-storage-at-the-edge#deploying-the-central-site
The DCN cross-site backup/restore feature requires all sites use a common "openstack" cephx client name, and that capability is addressed by bug #1857776. Therefore, I switched the component of this BZ simply so I could reference the same fixed-in-version. As noted in the BZ description, the cinder-backup service is deployed on the central site (not at DCN sites). The "cinder" cli is required to perform a cross-AZ backup because it requires an API microversion. The "openstack" cli will not work because it currently doesn't support cinder microversions. A suitable backup command will look like this, where "vol-dcn1" is the name of a volume on the dcn1 site: % OS_VOLUME_API_VERSION=3.51 cinder backup-create --name vol-dcn1 --availability-zone az-central vol-dcn1 The backup can be restored to any site using cinder's ability to create volumes from a backup in any AZ. Here are commands that would essentially restore the backup to a new "vol-dcn2" volume located in the dcn2 site. % BACKUP_ID=$(openstack volume backup list --name vol-dcn1 -f value -c ID) % OS_VOLUME_API_VERSION=3.51 cinder create --availability-zone az-dcn2 --name vol-dcn2 --backup-id $BACKUP_ID 1 A good test case would be to use a bootable volume on dcn1. Create a backup, restore it to create a new volume on dcn2, and then verify you can boot an instance on dcn2 from the new volume.
According to our records, this should be resolved by python-tripleoclient-12.3.2-1.20200914164930.el8ost. This build is available now.
Moving to 16.1.5
If the dcn ceph keyrings are present on the controller's host, but not visible inside the containers that need them (in this instance, the cinder-backup container), then it probably means the container needs to be restarted (the keyrings are copied from the host to the container on startup). If the problem described in comment #11 is resolved by restarting the c-bak container, then we need to account for that in the documentation. In OSP-16.1 the associated documentation is [1] (Section 4.5. Updating the central location). That update will cause the glance-api service to be restarted because it affect the glance configuration (/etc/glance/glance-api.conf). But Step 4.5 does *not* modify cinder's configuration, so as far as TripleO is concerned the cinder-volume and cinder-backup services don't need to be restarted. This is likely the source of this bug (BTW, c-vol will neeed to be restarted or else offline volume migration from the central site -TO- an edge site won't work). [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/distributed_compute_node_and_storage_deployment/assembly_deploying-storage-at-the-edge#updating-the-central-location_deploy-with-storage So, restart the c-bak container on the central site and see if the backup operation works.
Confirming Alan's solution works, after restating c-backup C-backup now does have access to dcn1.client.openstack.keyring and thus backup works. [root@central-controller0-2 ~]# podman exec -it openstack-cinder-backup-podman-0 ls /etc/ceph central.client.admin.keyring central.client.crash.keyring central.client.openstack.keyring central.conf central.mon.keyring dcn1.client.openstack.keyring dcn1.conf dcn2.client.openstack.keyring dcn2.conf rbdmap (central) [stack@site-undercloud-0 ~]$ OS_VOLUME_API_VERSION=3.51 cinder backup-create --name vol-dcn1-retest --availability-zone az-central vol-dcn1 +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 61463051-09e7-4b29-bbae-fe640c12b46f | | name | vol-dcn1-retest | | volume_id | 675838a2-677a-4a7e-bd23-eb2543b4c954 | +-----------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ cinder backup-list +--------------------------------------+--------------------------------------+-----------+-----------------+------+--------------+-----------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+-----------------+------+--------------+-----------+ | 61463051-09e7-4b29-bbae-fe640c12b46f | 675838a2-677a-4a7e-bd23-eb2543b4c954 | available | vol-dcn1-retest | 2 | 0 | backups | +--------------------------------------+--------------------------------------+-----------+-----------------+------+--------------+-----------+ I'll retesting once more, including restore before officially verifying this bz. But it looks better.
Thanks, Tzach, but let's not overlook the need for a doc update (another bz?).
Doc bug reported: https://bugzilla.redhat.com/show_bug.cgi?id=1961663
Verfied on: python3-tripleoclient-12.3.2-1.20210407123431.ae58329.el8ost.noarch On a DCN ceph backed deployment, with c-backup installed on central site. We already had an exsiting cirros image uploaded: (central) [stack@site-undercloud-0 ~]$ glance image-list +--------------------------------------+----------------------------------+ | ID | Name | +--------------------------------------+----------------------------------+ | caa6ca92-24d6-4c3f-acdb-d09bb124bd63 | cirros-0.4.0-x86_64-disk.img | Using dcn1rc, lets boot an instances from a volume created from an image: (dcn1) [stack@site-undercloud-0 ~]$ nova boot --flavor m1.nano --block-device source=image,id=caa6ca92-24d6-4c3f-acdb-d09bb124bd63,dest=volume,size=2,shutdown=preserve,bootindex=0 SourceInstBootFromVol --availability-zone az-dcn1 --nic net-id=7b9c0db3-f7ab-4c66-a19a-1fa4b1a507db +--------------------------------------+-------------------------------------------------+ | Property | Value | +--------------------------------------+-------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | az-dcn1 | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | sourceinstbootfromvol | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-wucig3kn | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | qB26tua87Ymv | | config_drive | | | created | 2021-05-19T07:31:12Z | | description | - | | flavor:disk | 1 | | flavor:ephemeral | 0 | | flavor:extra_specs | {"hw_rng:allowed": "True"} | | flavor:original_name | m1.nano | | flavor:ram | 128 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | | | host_status | | | id | 2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d | | image | Attempt to boot from volume - no image supplied | | key_name | - | | locked | False | | locked_reason | - | | metadata | {} | | name | SourceInstBootFromVol | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | server_groups | [] | | status | BUILD | | tags | [] | | tenant_id | b32b24e1449e4accaa9cc047d184319d | | trusted_image_certificates | - | | updated | 2021-05-19T07:31:12Z | | user_id | 380b318d9e484780a01eafbd8ce4d653 | +--------------------------------------+-------------------------------------------------+ Instance is up, we see above it's booted on dcn1 -> az-dcn1 (dcn1) [stack@site-undercloud-0 ~]$ nova list +--------------------------------------+-----------------------+--------+------------+-------------+------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------+--------+------------+-------------+------------------------------------------+ | 2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d | SourceInstBootFromVol | ACTIVE | - | Running | heat_tempestconf_network=192.168.199.159 | +--------------------------------------+-----------------------+--------+------------+-------------+------------------------------------------+ (dcn1) [stack@site-undercloud-0 ~]$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | 0a767a25-63d1-4c19-a966-7dd9de860f75 | in-use | | 2 | tripleo | true | 2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d | +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ It's bootable volume is also created on az-dcn1 (dcn1) [stack@site-undercloud-0 ~]$ cinder show 0a767a25-63d1-4c19-a966-7dd9de860f75 +--------------------------------+-------------------------------------------------+ | Property | Value | +--------------------------------+-------------------------------------------------+ | attached_servers | ['2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d'] | | attachment_ids | ['e9c27d37-9418-4635-9705-c2e7b7654306'] | | availability_zone | az-dcn1 | | bootable | true | | consistencygroup_id | None | | created_at | 2021-05-19T07:31:25.000000 | | description | | | encrypted | False | | id | 0a767a25-63d1-4c19-a966-7dd9de860f75 | | metadata | | | migration_status | None | | multiattach | False | | name | | | os-vol-host-attr:host | dcn1-computehci1-0@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | b32b24e1449e4accaa9cc047d184319d | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | None | | status | in-use | | updated_at | 2021-05-19T07:31:44.000000 | | user_id | 380b318d9e484780a01eafbd8ce4d653 | | volume_image_metadata | checksum : ba3cd24377dde5dfdd58728894004abb | | | container_format : bare | | | disk_format : raw | | | image_id : caa6ca92-24d6-4c3f-acdb-d09bb124bd63 | | | image_name : cirros-0.4.0-x86_64-disk.img | | | min_disk : 0 | | | min_ram : 0 | | | os_glance_failed_import : | | | os_glance_importing_to_stores : | | | size : 46137344 | | volume_type | tripleo | +--------------------------------+-------------------------------------------------+ Now lets backup said boot volume to central site: (central) [stack@site-undercloud-0 ~]$ OS_VOLUME_API_VERSION=3.51 cinder backup-create --name vol-dcn1-backup --availability-zone az-central 0a767a25-63d1-4c19-a966-7dd9de860f75 --force +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 95d1981b-942e-4726-ba8a-70578a524d2a | | name | vol-dcn1-backup | | volume_id | 0a767a25-63d1-4c19-a966-7dd9de860f75 | +-----------+--------------------------------------+ Backup is created which is good: (central) [stack@site-undercloud-0 ~]$ OS_VOLUME_API_VERSION=3.51 cinder backup-create --name vol-dcn1-backup --availability-zone az-central 0a767a25-63d1-4c19-a966-7dd9de860f75 --force +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 95d1981b-942e-4726-ba8a-70578a524d2a | | name | vol-dcn1-backup | | volume_id | 0a767a25-63d1-4c19-a966-7dd9de860f75 | +-----------+--------------------------------------+ Backup is completed: (central) [stack@site-undercloud-0 ~]$ cinder backup-list +--------------------------------------+--------------------------------------+-----------+-----------------+------+--------------+-----------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+-----------------+------+--------------+-----------+ | 95d1981b-942e-4726-ba8a-70578a524d2a | 0a767a25-63d1-4c19-a966-7dd9de860f75 | available | vol-dcn1-backup | 2 | 0 | backups | +--------------------------------------+--------------------------------------+-----------+-----------------+------+--------------+-----------+ But lets also restore data to confirm backup worked as expected. Create an empty restore target volume on DCN2: (dcn2) [stack@site-undercloud-0 ~]$ cinder create 2 --name restore-vol --availability-zone az-dcn2 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | az-dcn2 | | bootable | false | | consistencygroup_id | None | | created_at | 2021-05-19T07:41:05.000000 | | description | None | | encrypted | False | | id | db12090c-3006-42de-bd7e-69dd0b088a7e | | metadata | {} | | migration_status | None | | multiattach | False | | name | restore-vol | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | b32b24e1449e4accaa9cc047d184319d | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | 380b318d9e484780a01eafbd8ce4d653 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (dcn2) [stack@site-undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 0a767a25-63d1-4c19-a966-7dd9de860f75 | in-use | | 2 | tripleo | true | 2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d | | db12090c-3006-42de-bd7e-69dd0b088a7e | available | restore-vol | 2 | tripleo | false | | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ Now lets restore Cinder backup from central to target volume on DCN2: (central) [stack@site-undercloud-0 ~]$ cinder backup-restore vol-dcn1-backup --volume restore-vol +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | backup_id | 95d1981b-942e-4726-ba8a-70578a524d2a | | volume_id | db12090c-3006-42de-bd7e-69dd0b088a7e | | volume_name | restore-vol | +-------------+--------------------------------------+ Now lets boot a second instances this time on DCN2 from the restored volume: (dcn2) [stack@site-undercloud-0 ~]$ nova boot --flavor m1.nano --block-device source=volume,id=db12090c-3006-42de-bd7e-69dd0b088a7e,dest=volume,size=2,shutdown=preserve,bootindex=0 RestoredVolInstance --availability-zone az-dcn2 --nic net-id=7b9c0db3-f7ab-4c66-a19a-1fa4b1a507db +--------------------------------------+-------------------------------------------------+ | Property | Value | +--------------------------------------+-------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | az-dcn2 | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | restoredvolinstance | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-wv9hdm02 | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | 4XaLQBwpDGsE | | config_drive | | | created | 2021-05-19T07:45:53Z | | description | - | | flavor:disk | 1 | | flavor:ephemeral | 0 | | flavor:extra_specs | {"hw_rng:allowed": "True"} | | flavor:original_name | m1.nano | | flavor:ram | 128 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | | | host_status | | | id | d5b3b800-3155-479d-9a00-fe6af7a8d4c8 | | image | Attempt to boot from volume - no image supplied | | key_name | - | | locked | False | | locked_reason | - | | metadata | {} | | name | RestoredVolInstance | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | server_groups | [] | | status | BUILD | | tags | [] | | tenant_id | b32b24e1449e4accaa9cc047d184319d | | trusted_image_certificates | - | | updated | 2021-05-19T07:45:53Z | | user_id | 380b318d9e484780a01eafbd8ce4d653 | +--------------------------------------+-------------------------------------------------+ We have both instances up and running, verfiying both backup from DCN1 to central, and restore from Central to DCN2 works as expected, good to verify. (dcn2) [stack@site-undercloud-0 ~]$ nova list +--------------------------------------+-----------------------+--------+------------+-------------+------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------+--------+------------+-------------+------------------------------------------+ | d5b3b800-3155-479d-9a00-fe6af7a8d4c8 | RestoredVolInstance | ACTIVE | - | Running | heat_tempestconf_network=192.168.199.114 | | 2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d | SourceInstBootFromVol | ACTIVE | - | Running | heat_tempestconf_network=192.168.199.159 | +--------------------------------------+-----------------------+--------+------------+-------------+------------------------------------------+ (dcn2) [stack@site-undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 0a767a25-63d1-4c19-a966-7dd9de860f75 | in-use | | 2 | tripleo | true | 2a20d2ad-be1a-4007-a5a3-7f2f3bc19f1d | | db12090c-3006-42de-bd7e-69dd0b088a7e | in-use | restore-vol | 2 | tripleo | true | d5b3b800-3155-479d-9a00-fe6af7a8d4c8 | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+