This bug was initially created as a copy of Bug #1882054 Offline volume migration between central and DCN sites should work as long as all sites use a common "openstack" cephx client (offline migration directly between two DCN sites is not supported). Here is an example of how to migrate a volume from dcn1 to central. The "openstack" cli can be used because no API microversion is required. % openstack volume migrate vol-dcn1 --host hostgroup@tripleo_ceph The same volume can be migrated from central to another DCN site by selecting one of the cinder-volume services running at that site (there will be 3 such services because cinder runs A/A at DCN sites). Use "openstack volume service list" to see the available choices. % openstack volume migrate vol-dcn1 --host dcn2-computehci2-0@tripleo_ceph
According to our records, this should be resolved by python-tripleoclient-12.3.2-1.20200914164930.el8ost. This build is available now.
Moving to 16.1.5
Hit new dependency issue, bz1962304 If however I re-apply John's workaround #9 step 3 on all dcn compute nodes and retest it works. Meaning once bz1962304 is fixed workaround won't be needed and this bz could be retested easily. (central) [stack@site-undercloud-0 ~]$ cinder create 1 --name test8 --availability-zone az-dcn2 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | az-dcn2 | | bootable | false | | consistencygroup_id | None | | created_at | 2021-05-19T19:37:28.000000 | | description | None | | encrypted | False | | id | 8bc4462b-9e54-491b-be6d-b19c89665aab | | metadata | {} | | migration_status | None | | multiattach | False | | name | test8 | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | b32b24e1449e4accaa9cc047d184319d | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | 380b318d9e484780a01eafbd8ce4d653 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ cinder migrate test8 hostgroup@tripleo_ceph Request to migrate volume 8bc4462b-9e54-491b-be6d-b19c89665aab has been accepted. (central) [stack@site-undercloud-0 ~]$ cinder show test8 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attached_servers | [] | | attachment_ids | [] | | availability_zone | az-central | --> vol migrated from az-dcn2 to central. | bootable | false | | consistencygroup_id | None | | created_at | 2021-05-19T19:37:28.000000 | | description | None | | encrypted | False | | id | 8bc4462b-9e54-491b-be6d-b19c89665aab | | metadata | | | migration_status | success | | multiattach | False | | name | test8 | | os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | success | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | b32b24e1449e4accaa9cc047d184319d | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | updated_at | 2021-05-19T19:40:02.000000 | | user_id | 380b318d9e484780a01eafbd8ce4d653 | | volume_type | tripleo | +--------------------------------+--------------------------------------+
Verified on: python3-tripleoclient-12.5.1-2.20210603180733.95feb7c.el8ost.1.noarch Deploy and DCN HCI system. 1. Lets boot an instance from an image, create bootable vol on DCN1 site: (dcn1) [stack@site-undercloud-0 ~]$ nova boot --flavor m1.nano --block-device source=image,id=38da2f3f-f82d-454e-84db-8e206c7ebe4c,dest=volume,size=2,shutdown=preserve,bootindex=0 SourceInstBootFromVol --availability-zone az-dcn1 --nic net-id=dc934d9c-8901-49fa-a0ab-d49b46c30137 +--------------------------------------+-------------------------------------------------+ | Property | Value | +--------------------------------------+-------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | az-dcn1 | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | sourceinstbootfromvol | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-a30sfjif | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | cMv68h8DeaWT | | config_drive | | | created | 2021-11-01T08:30:42Z | | description | - | | flavor:disk | 1 | | flavor:ephemeral | 0 | | flavor:extra_specs | {"hw_rng:allowed": "True"} | | flavor:original_name | m1.nano | | flavor:ram | 128 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | | | host_status | | | id | 71d89e65-44ee-4e75-9508-b6b6b24b7950 | | image | Attempt to boot from volume - no image supplied | | key_name | - | | locked | False | | locked_reason | - | | metadata | {} | | name | SourceInstBootFromVol | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | server_groups | [] | | status | BUILD | | tags | [] | | tenant_id | 63f10193b44a487383ab94edd94ce70f | | trusted_image_certificates | - | | updated | 2021-11-01T08:30:42Z | | user_id | ffac91840c5c4dc5b38f10315cbef7ce | +--------------------------------------+-------------------------------------------------+ 2. Instance is up, a new bootable volume is created/in-use: (dcn1) [stack@site-undercloud-0 ~]$ nova list cinder list +--------------------------------------+-----------------------+--------+------------+-------------+-----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------+--------+------------+-------------+-----------------------+ | 71d89e65-44ee-4e75-9508-b6b6b24b7950 | SourceInstBootFromVol | ACTIVE | - | Running | internal=192.168.0.27 | +--------------------------------------+-----------------------+--------+------------+-------------+-----------------------+ (dcn1) [stack@site-undercloud-0 ~]$ cinder list +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | in-use | | 2 | tripleo | true | 71d89e65-44ee-4e75-9508-b6b6b24b7950 | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ 3. Using central site's rc lets backup the cinder volume: (central) [stack@site-undercloud-0 ~]$ OS_VOLUME_API_VERSION=3.51 cinder backup-create --name vol-dcn1-backupA --availability-zone az-central d56a43aa-3a7a-4c1c-bf49-8629329f4d17 --force +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 5fd15d7a-3d98-4924-811a-bf35503d2fac | | name | vol-dcn1-backupA | | volume_id | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | +-----------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ cinder backup-list +--------------------------------------+--------------------------------------+--------+------------------+------+--------------+-----------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+--------+------------------+------+--------------+-----------+ | 5fd15d7a-3d98-4924-811a-bf35503d2fac | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | error | vol-dcn1-backupA | 2 | 0 | - | +--------------------------------------+--------------------------------------+--------+------------------+------+--------------+-----------+ Now again as we had hit on the 16.1 bz https://bugzilla.redhat.com/show_bug.cgi?id=1882054#c12 C-backup log reports same error: /var/log/containers/cinder/cinder-backup.log:581:2021-11-01 08:32:47.233 36 ERROR oslo_messaging.rpc.server Traceback (most recent call last): /var/log/containers/cinder/cinder-backup.log:582:2021-11-01 08:32:47.233 36 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 79, in _check_or_get_keyring_contents /var/log/containers/cinder/cinder-backup.log:583:2021-11-01 08:32:47.233 36 ERROR oslo_messaging.rpc.server with open(keyring_path, 'r') as keyring_file: /var/log/containers/cinder/cinder-backup.log:584:2021-11-01 08:32:47.233 36 ERROR oslo_messaging.rpc.server FileNotFoundError: [Errno 2] No such file or directory: '/etc/ceph/dcn1.client.openstack.keyring' /var/log/containers/cinder/cinder-backup.log:585:2021-11-01 08:32:47.233 36 ERROR oslo_messaging.rpc.server .. Lets restart cinder backup for it to get access to the relevant keys [root@central-controller0-2 ~]# podman exec -it openstack-cinder-backup-podman-0 ls /etc/ceph central.client.admin.keyring central.client.crash.keyring central.client.openstack.keyring central.conf central.mon.keyring rbdmap ^ notice missing "dcn1.client.openstack.keyring " [root@central-controller0-2 ceph]# podman restart openstack-cinder-backup-podman-0 5deb0657915cd439bc57c737f6d6dcf34290fc187ef5b2f0080238dbea3be591 [root@central-controller0-2 ceph]# podman exec -it openstack-cinder-backup-podman-0 /bin/bash [root@central-controller0-2 ~]# podman exec -it openstack-cinder-backup-podman-0 ls /etc/ceph central.client.admin.keyring central.mon.keyring dcn2.conf central.client.crash.keyring dcn1.client.openstack.keyring rbdmap central.client.openstack.keyring dcn1.conf central.conf dcn2.client.openstack.keyring Now we backup has access to all the keys 4. (central) [stack@site-undercloud-0 ~]$ OS_VOLUME_API_VERSION=3.51 cinder backup-create --name vol-dcn1-backupB --availability-zone az-central d56a43aa-3a7a-4c1c-bf49-8629329f4d17 --force +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | d11ed44b-7db6-4f93-9ac0-33a77c73ccb2 | | name | vol-dcn1-backupB | | volume_id | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | +-----------+--------------------------------------+ 5. Backup works this time around: (central) [stack@site-undercloud-0 ~]$ cinder backup-list +--------------------------------------+--------------------------------------+-----------+------------------+------+--------------+-----------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+------------------+------+--------------+-----------+ | 5fd15d7a-3d98-4924-811a-bf35503d2fac | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | error | vol-dcn1-backupA | 2 | 0 | - | | d11ed44b-7db6-4f93-9ac0-33a77c73ccb2 | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | available | vol-dcn1-backupB | 2 | 0 | backups | +--------------------------------------+--------------------------------------+-----------+------------------+------+--------------+-----------+ Lets ignore the fact that I had to restart cinder backup, this is related to deployment/documentation issue. I've cloned the 16.1 doc bz for 16.2 here: https://bugzilla.redhat.com/show_bug.cgi?id=2018924 6. Lets also verify that restoring data dcn2 site works: (dcn2) [stack@site-undercloud-0 ~]$ cinder create 2 --name restore-vol --availability-zone az-dcn2 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | az-dcn2 | | bootable | false | | consistencygroup_id | None | | created_at | 2021-11-01T09:36:42.000000 | | description | None | | encrypted | False | | id | 1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6 | | metadata | {} | | migration_status | None | | multiattach | False | | name | restore-vol | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 63f10193b44a487383ab94edd94ce70f | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | ffac91840c5c4dc5b38f10315cbef7ce | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (dcn2) [stack@site-undercloud-0 ~]$ cinder show 1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6 +--------------------------------+----------------------------------------------+ | Property | Value | +--------------------------------+----------------------------------------------+ | attached_servers | [] | | attachment_ids | [] | | availability_zone | az-dcn2 | | bootable | false | | consistencygroup_id | None | | created_at | 2021-11-01T09:36:42.000000 | | description | None | | encrypted | False | | id | 1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6 | | metadata | | | migration_status | None | | multiattach | False | | name | restore-vol | | os-vol-host-attr:host | dcn2-computehci2-2@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 63f10193b44a487383ab94edd94ce70f | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | None | | status | available | | updated_at | 2021-11-01T09:36:45.000000 | | user_id | ffac91840c5c4dc5b38f10315cbef7ce | | volume_type | tripleo | +--------------------------------+----------------------------------------------+ (central) [stack@site-undercloud-0 ~]$ cinder backup-restore vol-dcn1-backupB --volume restore-vol +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | backup_id | d11ed44b-7db6-4f93-9ac0-33a77c73ccb2 | | volume_id | 1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6 | | volume_name | restore-vol | +-------------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ cinder list +--------------------------------------+------------------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+------------------+-------------+------+-------------+----------+--------------------------------------+ | 1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6 | restoring-backup | restore-vol | 2 | tripleo | false | | | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | in-use | | 2 | tripleo | true | 71d89e65-44ee-4e75-9508-b6b6b24b7950 | +--------------------------------------+------------------+-------------+------+-------------+----------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6 | available | restore-vol | 2 | tripleo | true | | | d56a43aa-3a7a-4c1c-bf49-8629329f4d17 | in-use | | 2 | tripleo | true | 71d89e65-44ee-4e75-9508-b6b6b24b7950 | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ 7. Final restore verification step, boot an instance from restored volume (dcn2) [stack@site-undercloud-0 ~]$ nova boot --flavor m1.nano --block-device source=volume,id=1db07156-7ceb-4220-bbeb-6cc2e8e0d6f6,dest=volume,size=2,shutdown=preserve,bootindex=0 RestoredVolInstance --availability-zone az-dcn2 --nic net-id=dc934d9c-8901-49fa-a0ab-d49b46c30137 +--------------------------------------+-------------------------------------------------+ | Property | Value | +--------------------------------------+-------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | az-dcn2 | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | restoredvolinstance | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-9t9goipa | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | 3Avmg8pT7xWw | | config_drive | | | created | 2021-11-01T09:44:51Z | | description | - | | flavor:disk | 1 | | flavor:ephemeral | 0 | | flavor:extra_specs | {"hw_rng:allowed": "True"} | | flavor:original_name | m1.nano | | flavor:ram | 128 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | | | host_status | | | id | 0b96425a-ec7b-4b87-8a45-9db8bfe177c4 | | image | Attempt to boot from volume - no image supplied | | key_name | - | | locked | False | | locked_reason | - | | metadata | {} | | name | RestoredVolInstance | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | server_groups | [] | | status | BUILD | | tags | [] | | tenant_id | 63f10193b44a487383ab94edd94ce70f | | trusted_image_certificates | - | | updated | 2021-11-01T09:44:51Z | | user_id | ffac91840c5c4dc5b38f10315cbef7ce | +--------------------------------------+-------------------------------------------------+ Yay both instances are up source (backup/dcn1)_and target(restore/dcn2) (dcn2) [stack@site-undercloud-0 ~]$ nova list +--------------------------------------+-----------------------+--------+------------+-------------+-----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------+--------+------------+-------------+-----------------------+ | 0b96425a-ec7b-4b87-8a45-9db8bfe177c4 | RestoredVolInstance | ACTIVE | - | Running | internal=192.168.0.20 | | 71d89e65-44ee-4e75-9508-b6b6b24b7950 | SourceInstBootFromVol | ACTIVE | - | Running | internal=192.168.0.27 | +--------------------------------------+-----------------------+--------+------------+-------------+-----------------------+ [root@dcn2-computehci2-2 /]# virsh console instance-0000016a Connected to domain 'instance-0000016a' Escape character is ^] (Ctrl + ]) login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root. restoredvolinstance login: cirros Password: $ #Inside recovered instance's console Thus proving that we managed to backup a volume from DCN1 to central site, then use restore/create a new volume from central site's backup to DCN2, Good to verify! Plus duplicating doc bug for 16.2.
Sorry, have to retest! I checked backup/restore while I actually needed to test offline vol migration. Got thrown off track by "This bug was initially created as a copy of Bug #1882054"
Update I can't verify this just yet despite ON_QA status, As DCN offline vol migration depends on another bz: https://bugzilla.redhat.com/show_bug.cgi?id=1962304
Still awaiting bz1962304 to reach ON_QA, before verifying this bz.
Retested the steps of #c11 (with OSC) and migration from central to dcn. The environment was deployed with openstack-tripleo-heat-templates-11.6.1-2.20220116004910.el8ost rhosp-director-images-x86_64-16.2-20220217.1.el8ost (central) [stack@site-undercloud-0 ~]$ openstack volume service list +------------------+------------------------------------+------------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+------------------------------------+------------+---------+-------+----------------------------+ | cinder-scheduler | central-controller0-2.redhat.local | az-central | enabled | up | 2022-03-02T10:07:53.000000 | | cinder-scheduler | central-controller0-1.redhat.local | az-central | enabled | up | 2022-03-02T10:07:56.000000 | | cinder-scheduler | central-controller0-0.redhat.local | az-central | enabled | up | 2022-03-02T10:07:52.000000 | | cinder-backup | central-controller0-2.redhat.local | az-central | enabled | up | 2022-03-02T10:07:50.000000 | | cinder-volume | hostgroup@tripleo_ceph | az-central | enabled | up | 2022-03-02T10:07:54.000000 | | cinder-volume | dcn1-compute1-0@tripleo_ceph | az-dcn1 | enabled | up | 2022-03-02T10:07:50.000000 | | cinder-volume | dcn1-compute1-1@tripleo_ceph | az-dcn1 | enabled | up | 2022-03-02T10:07:55.000000 | | cinder-volume | dcn2-computehci2-0@tripleo_ceph | az-dcn2 | enabled | up | 2022-03-02T10:07:52.000000 | | cinder-volume | dcn2-computehci2-1@tripleo_ceph | az-dcn2 | enabled | up | 2022-03-02T10:07:48.000000 | | cinder-volume | dcn2-computehci2-2@tripleo_ceph | az-dcn2 | enabled | up | 2022-03-02T10:07:49.000000 | +------------------+------------------------------------+------------+---------+-------+----------------------------+ (central) [stack@site-undercloud-0 ~]$ openstack volume create --size 1 --availability-zone az-dcn1 testosc1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | az-dcn1 | | bootable | false | | consistencygroup_id | None | | created_at | 2022-03-02T10:07:58.000000 | | description | None | | encrypted | False | | id | f2718e98-fa2b-4f1a-a4d9-19b29e8381b9 | | migration_status | None | | multiattach | False | | name | testosc1 | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | type | tripleo | | updated_at | None | | user_id | 75476ad47c01489eb8fe6e43178ab478 | +---------------------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ openstack volume create --size 1 --availability-zone az-dcn2 testosc2 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | az-dcn2 | | bootable | false | | consistencygroup_id | None | | created_at | 2022-03-02T10:08:00.000000 | | description | None | | encrypted | False | | id | 6914e1bc-0547-4f6f-bdb4-a432395adafa | | migration_status | None | | multiattach | False | | name | testosc2 | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | type | tripleo | | updated_at | None | | user_id | 75476ad47c01489eb8fe6e43178ab478 | +---------------------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ openstack volume migrate testosc1 --host hostgroup@tripleo_ceph (central) [stack@site-undercloud-0 ~]$ openstack volume migrate testosc2 --host hostgroup@tripleo_ceph --> Wait a bit, then: (central) [stack@site-undercloud-0 ~]$ openstack volume show -c availability_zone testosc1 +-------------------+------------+ | Field | Value | +-------------------+------------+ | availability_zone | az-central | +-------------------+------------+ (central) [stack@site-undercloud-0 ~]$ openstack volume show -c availability_zone testosc2 +-------------------+------------+ | Field | Value | +-------------------+------------+ | availability_zone | az-central | +-------------------+------------+ --> Migrate back: (central) [stack@site-undercloud-0 ~]$ openstack volume migrate testosc1 --host dcn2-computehci2-2@tripleo_ceph (central) [stack@site-undercloud-0 ~]$ openstack volume migrate testosc2 --host dcn1-compute1-0@tripleo_ceph --> Wait a bit, finally: (central) [stack@site-undercloud-0 ~]$ openstack volume show -c availability_zone testosc1 +-------------------+---------+ | Field | Value | +-------------------+---------+ | availability_zone | az-dcn2 | +-------------------+---------+ (central) [stack@site-undercloud-0 ~]$ openstack volume show -c availability_zone testosc2 +-------------------+---------+ | Field | Value | +-------------------+---------+ | availability_zone | az-dcn1 | +-------------------+---------+