Bug 1472723
| Summary: | Nova resize fails despite of openstack-nova-migration package being present on the compute nodes | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Punit Kundal <pkundal> | |
| Component: | openstack-nova | Assignee: | Ollie Walsh <owalsh> | |
| Status: | CLOSED ERRATA | QA Contact: | Archit Modi <amodi> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 7.0 (Kilo) | CC: | amodi, awaugama, berrange, cshastri, dasmith, dmacpher, eglynn, jraju, kchamart, mbooth, mschuppe, owalsh, pneedle, sbauza, sclewis, sferdjao, sgordon, srevivo, vromanso | |
| Target Milestone: | zstream | Keywords: | Triaged, ZStream | |
| Target Release: | 7.0 (Kilo) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-nova-2015.1.4-45.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
This fix updates the whitelist for certain SSH commands used during instance migration.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1481389 1481393 (view as bug list) | Environment: | ||
| Last Closed: | 2017-10-19 18:19:51 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1267598, 1474982, 1481389, 1481393 | |||
I just checked resizing an instance on OSP10 setup without setting up ssh tunneling, and the resize works without any issue. The issue here is that we're not whitelisting the correct scp command for OSP8. We can see from the logs (if you squint through the capitalisation) that the executed command is: SCP /VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123_RESIZE/DISK 192.0.2.21:/VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123/DISK The nova_migration user is configured to pass ssh commands through nova-migration-wrapper, installed by openstack-nova-migration. This whitelists: ['scp', '-r', '-t'] This fails because the '-r', and '-t' options are missing, and the wrapper responds 'Forbidden'. The -r argument has been present since version 14 (Newton, OSP 10). I can't see that the -t argument has ever been present, so either I'm looking in the wrong place or this has never worked for any version of OSP. We need to fix the whitelisting in nova-migration-wrapper, and update it appropriately for all OSP versions. Ignore the -t option, it's the hidden option that scp appends on the target side. i.e to allow scp <foo> <bar> we need to whitelist scp -t .... on the migration target host. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2994 |
Description of problem: As per [1]. The openstack-nova-migration package ensures that the necessary configuration required for migrating instances is correctly setup. I have the below version of openstack-nova-migration package present on my compute nodes: [root@overcloud-compute-1 ~]# rpm -qa | grep openstack-nova-migration openstack-nova-migration-12.0.6-13.el7ost.noarch Despite of this package being present on the compute node nodes, the migration/resize operations are failing. Version-Release number of selected component (if applicable): openstack-nova-migration-12.0.6-13.el7ost.noarch How reproducible: Always Steps to Reproduce: 1. Deploy RHOSP 8 env via director with a minimum of 2 compute nodes 2. Ensure the openstack-nova-migration package is present on the compute nodes 3. Try to migrate/resize an instance it fails. Here are my steps: Initially, I launched an instance and tried to resize it directly, this ended in an error: +++ [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b | test1 | ACTIVE | - | Running | net1=172.168.122.3 | +--------------------------------------+-------+--------+------------+-------------+--------------------+ [stack@instack ~(overcloud_admin)]$ nova show test1 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCFiskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-compute-1.localdomain | | OS-EXT-SRV-ATTR:hostname | test1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-1.localdomain | [stack@instack ~(overcloud_admin)]$ openstack flavor list ...... | 6 | m1.custom | 1024 | 1 | 0 | 1 | True | [stack@instack ~(overcloud_admin)]$ openstack server resize --flavor m1.custom test1 [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b | test1 | ERROR | - | Running | net1=172.168.122.3 | +--------------------------------------+-------+--------+------------+-------------+--------------------+ +++ After checking the logs on the compute node, I found that the error was hit because the instance disk could not be copied to another node (please look at the log lines in capital letters): +++ [root@overcloud-compute-1 ~]# grep 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b /var/log/nova/nova-compute.log | grep ERROR 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [req-6f7ffa18-e9f0-41c4-989b-bb44066ee56b 5494dff29f5d4092a7087da043642406 fac27b116b34423d9d6d4425a9cd218c - - -] [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] Setting instance vm_state to ERROR 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] Traceback (most recent call last): 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6456, in _error_out_instance_on_exception 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] yield 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3852, in resize_instance 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] timeout, retry_interval) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6988, in migrate_disk_and_power_off 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] shared_storage) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] six.reraise(self.type_, self.value, self.tb) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6973, in migrate_disk_and_power_off 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] compression=compression) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 220, in copy_image 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] compression=compression) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 124, in copy_file 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] compression=compression) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 209, in copy_file 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] on_execute=on_execute, on_completion=on_completion) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] return processutils.execute(*cmd, **kwargs) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 342, in execute 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] cmd=sanitized_cmd) 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] ProcessExecutionError: Unexpected error while running command. 2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] COMMAND: SCP /VAR/LIB/NOVA/INSTANCES/49C41871-6EA8-4FA6-A7BF-5FD52A6D095B_RESIZE/DISK 192.0.2.21:/VAR/LIB/NOVA/INSTANCES/49C41871-6EA8-4FA6-A7BF-5FD52A6D095B/DISK 2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] EXIT CODE: 1 2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] STDOUT: U'' 2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] STDERR: U'FORBIDDEN\NLOST CONNECTION\N' 2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] 2017-07-11 13:52:18.098 27294 ERROR oslo_messaging.rpc.dispatcher Command: scp /var/lib/nova/instances/49c41871-6ea8-4fa6-a7bf-5fd52a6d095b_resize/disk 192.0.2.21:/var/lib/nova/instances/49c41871-6ea8-4fa6-a7bf-5fd52a6d095b/disk +++ Looking at the documentation here [2] , I can see that in order to resize/migrate instances, we need to have ssh tunneling setup between the compute nodes. After following this procedure, I again to resize an instance, it failed with the same error as before: +++ stack@instack ~(overcloud_admin)]$ openstack server resize --flavor m1.custom test2 [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123 | test2 | ERROR | - | Running | net1=172.168.122.4 | +--------------------------------------+-------+--------+------------+-------------+--------------------+ [root@overcloud-compute-1 ~]# grep ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123 /var/log/nova/nova-compute.log | grep ERROR 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [req-0185e9b8-0fcc-4caf-8f0d-caf24e066b4a 5494dff29f5d4092a7087da043642406 fac27b116b34423d9d6d4425a9cd218c - - -] [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] Setting instance vm_state to ERROR 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] Traceback (most recent call last): 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6456, in _error_out_instance_on_exception 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] yield 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3852, in resize_instance 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] timeout, retry_interval) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6988, in migrate_disk_and_power_off 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] shared_storage) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] six.reraise(self.type_, self.value, self.tb) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6973, in migrate_disk_and_power_off 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] compression=compression) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 220, in copy_image 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] compression=compression) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 124, in copy_file 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] compression=compression) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 209, in copy_file 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] on_execute=on_execute, on_completion=on_completion) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] return processutils.execute(*cmd, **kwargs) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 342, in execute 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] cmd=sanitized_cmd) 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] ProcessExecutionError: Unexpected error while running command. 2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] COMMAND: SCP /VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123_RESIZE/DISK 192.0.2.21:/VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123/DISK 2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] EXIT CODE: 1 2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] STDOUT: U'' 2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] STDERR: U'FORBIDDEN\NLOST CONNECTION\N' 2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] 2017-07-11 14:10:15.807 27294 ERROR oslo_messaging.rpc.dispatcher Command: scp /var/lib/nova/instances/ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123_resize/disk 192.0.2.21:/var/lib/nova/instances/ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123/disk +++ The error was basically the same, scp failure (line in capital letters here as well) I performed a few more checks and found that nova user was not able to ssh into the other compute node as nova user even after configuring password less access by following the steps at [2]: +++ bash-4.2$ ssh 192.0.2.19 This command must be run via SSH ForceCommand (see man 5 sshd_config). Connection to 192.0.2.19 closed. -bash-4.2$ ssh 192.0.2.21 This command must be run via SSH ForceCommand (see man 5 sshd_config). Connection to 192.0.2.21 closed. +++ After a little more digging, I found that the file "/var/lib/nova/.ssh/config" is coming from the openstack-nova-migration on the compute nodes: [root@overcloud-compute-1 ~]# rpm -ql openstack-nova-migration-12.0.6-13.el7ost /etc/nova/migration /etc/nova/migration/authorized_keys /etc/nova/migration/identity /etc/nova/migration/rootwrap.conf /etc/nova/migration/rootwrap.d /etc/nova/migration/rootwrap.d/cold_migration.filters /etc/sudoers.d/nova_migration /usr/bin/nova-migration-wrapper /var/lib/nova/.ssh /var/lib/nova/.ssh/config +++ Now by default, this is what is present in the "/var/lib/nova/.ssh/config" file: +++ bash-4.2$ cat .ssh/config Host * User nova_migration UserKnownHostsFile /dev/null IdentityFile /etc/nova/migration/identity +++ Now once I commented out these lines in the "/var/lib/nova/.ssh/config" file on both the compute nodes as below: +++ -bash-4.2$ cat .ssh/config Host * # User nova_migration # UserKnownHostsFile /dev/null # IdentityFile /etc/nova/migration/identity +++ I found that nova user was able to ssh into the other compute node as nova user +++ -bash-4.2$ whoami nova -bash-4.2$ ssh 192.0.2.19 Last login: Tue Jul 11 14:15:44 2017 from overcloud-compute-0.localdomain -bash-4.2$ hostname overcloud-compute-1.localdomain -bash-4.2$ whoami nova +++ Once this was done, I launched another instance and tried to resize it again and it worked fine this time: +++ [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | ACTIVE | - | Running | net1=172.168.122.5 | +--------------------------------------+-------+--------+------------+-------------+--------------------+ [stack@instack ~(overcloud_admin)]$ nova show test3 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCFiskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-compute-1.localdomain | < = initially launched on compute-1 | OS-EXT-SRV-ATTR:hostname | test3 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-1.localdomain | [stack@instack ~(overcloud_admin)]$ openstack server resize --flavor m1.custom test3 [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+------------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------------+-------------+--------------------+ | 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | RESIZE | resize_migrating | Running | net1=172.168.122.5 | +--------------------------------------+-------+--------+------------------+-------------+--------------------+ [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+---------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+---------------+-------------+--------------------+ | 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | RESIZE | resize_finish | Running | net1=172.168.122.5 | +--------------------------------------+-------+--------+---------------+-------------+--------------------+ [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+---------------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+---------------+------------+-------------+--------------------+ | 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | VERIFY_RESIZE | - | Running | net1=172.168.122.5 | +--------------------------------------+-------+---------------+------------+-------------+--------------------+ [stack@instack ~(overcloud_admin)]$ openstack server resize --confirm 74b33950-3dcd-4444-ba0e-20b08be35b09 [stack@instack ~(overcloud_admin)]$ nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | ACTIVE | - | Running | net1=172.168.122.5 | < = resized and disk moved to compute-0 +--------------------------------------+-------+--------+------------+-------------+--------------------+ [stack@instack ~(overcloud_admin)]$ nova show test3 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCFiskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-compute-0.localdomain | | OS-EXT-SRV-ATTR:hostname | test3 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-0.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | [root@overcloud-compute-0 ~]# virsh list Id Name State ---------------------------------------------------- 1 instance-00000003 running Surprisingly enough, it works when directly used with virsh commands: +++ [root@overcloud-compute-1 ~]# virsh -c qemu+ssh://nova_migration@overcloud-compute-0/system?keyfile=/etc/nova/migration/identity Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # list Id Name State ---------------------------------------------------- 1 instance-00000003 running virsh # +++ P.S I even tried setting "allow_resize_to_same_host" to "true" in nova.conf on the compute nodes, that failed too. It was only that after following the steps at [2] and then commenting out the above mentioned lines in "/var/lib/nova/.ssh/config" I was able to resize my instances. [1] https://access.redhat.com/downloads/content/openstack-nova-migration/2015.1.4-36.el7ost/noarch/fd431d51/package [2] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html-single/migrating_instances/#configure_ssh_tunneling_between_nodes Additional info: If the openstack-nova-migration package is taking care of