Bug 1472723 - Nova resize fails despite of openstack-nova-migration package being present on the compute nodes
Nova resize fails despite of openstack-nova-migration package being present o...
Status: ON_DEV
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: ---
: 7.0 (Kilo)
Assigned To: Ollie Walsh
Joe H. Rahme
: Triaged, ZStream
Depends On:
Blocks: 1481389 1481393
  Show dependency treegraph
 
Reported: 2017-07-19 05:23 EDT by Punit Kundal
Modified: 2017-08-15 10:40 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Deprecated Functionality
Doc Text:
The docs for manual ssh tunnel setup need to be removed (I thought they already had?) https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html-single/migrating_instances/#configure_ssh_tunneling_between_nodes
Story Points: ---
Clone Of:
: 1481389 1481393 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
RDO 8074 None None None 2017-08-02 07:15 EDT
RDO 8573 None None None 2017-08-15 10:40 EDT

  None (edit)
Description Punit Kundal 2017-07-19 05:23:27 EDT
Description of problem:

As per [1]. The openstack-nova-migration package ensures that the necessary configuration required for migrating instances is correctly setup. 

I have the below version of openstack-nova-migration package present on my compute nodes:

[root@overcloud-compute-1 ~]# rpm -qa | grep openstack-nova-migration
openstack-nova-migration-12.0.6-13.el7ost.noarch

Despite of this package being present on the compute node nodes, the migration/resize operations are failing. 

Version-Release number of selected component (if applicable):
openstack-nova-migration-12.0.6-13.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Deploy RHOSP 8 env via director with a minimum of 2 compute nodes
2. Ensure the openstack-nova-migration package is present on the compute nodes
3. Try to migrate/resize an instance it fails.


Here are my steps:

Initially, I launched an instance and tried to resize it directly, this ended in an error:

+++
[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks           |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b | test1 | ACTIVE | -          | Running     | net1=172.168.122.3 |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
[stack@instack ~(overcloud_admin)]$ nova show test1
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCFiskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | overcloud-compute-1.localdomain                          |
| OS-EXT-SRV-ATTR:hostname             | test1                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-compute-1.localdomain                          |

[stack@instack ~(overcloud_admin)]$ openstack flavor list
......
| 6  | m1.custom |  1024 |    1 |         0 |     1 | True      |

[stack@instack ~(overcloud_admin)]$ openstack server resize --flavor m1.custom test1

[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks           |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b | test1 | ERROR  | -          | Running     | net1=172.168.122.3 |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
+++

After checking the logs on the compute node, I found that the error was hit because the instance disk could not be copied to another node (please look at the log lines in capital letters):

+++
[root@overcloud-compute-1 ~]# grep 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b /var/log/nova/nova-compute.log | grep ERROR
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [req-6f7ffa18-e9f0-41c4-989b-bb44066ee56b 5494dff29f5d4092a7087da043642406 fac27b116b34423d9d6d4425a9cd218c - - -] [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] Setting instance vm_state to ERROR
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] Traceback (most recent call last):
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6456, in _error_out_instance_on_exception
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     yield
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3852, in resize_instance
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     timeout, retry_interval)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6988, in migrate_disk_and_power_off
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     shared_storage)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     six.reraise(self.type_, self.value, self.tb)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6973, in migrate_disk_and_power_off
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     compression=compression)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 220, in copy_image
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     compression=compression)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 124, in copy_file
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     compression=compression)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 209, in copy_file
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     on_execute=on_execute, on_completion=on_completion)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     return processutils.execute(*cmd, **kwargs)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 342, in execute
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b]     cmd=sanitized_cmd)
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] ProcessExecutionError: Unexpected error while running command.
2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] COMMAND: SCP /VAR/LIB/NOVA/INSTANCES/49C41871-6EA8-4FA6-A7BF-5FD52A6D095B_RESIZE/DISK 192.0.2.21:/VAR/LIB/NOVA/INSTANCES/49C41871-6EA8-4FA6-A7BF-5FD52A6D095B/DISK
2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] EXIT CODE: 1
2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] STDOUT: U''
2017-07-11 13:52:17.805 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: 49C41871-6EA8-4FA6-A7BF-5FD52A6D095B] STDERR: U'FORBIDDEN\NLOST CONNECTION\N'
2017-07-11 13:52:17.805 27294 ERROR nova.compute.manager [instance: 49c41871-6ea8-4fa6-a7bf-5fd52a6d095b] 
2017-07-11 13:52:18.098 27294 ERROR oslo_messaging.rpc.dispatcher Command: scp /var/lib/nova/instances/49c41871-6ea8-4fa6-a7bf-5fd52a6d095b_resize/disk 192.0.2.21:/var/lib/nova/instances/49c41871-6ea8-4fa6-a7bf-5fd52a6d095b/disk
+++


Looking at the documentation here [2] , I can see that in order to resize/migrate instances, we need to have ssh tunneling setup between the compute nodes. 

After following this procedure, I again to resize an instance, it failed with the same error as before:

+++
 stack@instack ~(overcloud_admin)]$ openstack server resize --flavor m1.custom test2

[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks           |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123 | test2 | ERROR  | -          | Running     | net1=172.168.122.4 |
+--------------------------------------+-------+--------+------------+-------------+--------------------+

[root@overcloud-compute-1 ~]# grep ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123 /var/log/nova/nova-compute.log | grep ERROR
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [req-0185e9b8-0fcc-4caf-8f0d-caf24e066b4a 5494dff29f5d4092a7087da043642406 fac27b116b34423d9d6d4425a9cd218c - - -] [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] Setting instance vm_state to ERROR
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] Traceback (most recent call last):
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6456, in _error_out_instance_on_exception
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     yield
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3852, in resize_instance
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     timeout, retry_interval)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6988, in migrate_disk_and_power_off
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     shared_storage)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     six.reraise(self.type_, self.value, self.tb)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6973, in migrate_disk_and_power_off
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     compression=compression)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 220, in copy_image
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     compression=compression)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 124, in copy_file
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     compression=compression)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 209, in copy_file
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     on_execute=on_execute, on_completion=on_completion)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     return processutils.execute(*cmd, **kwargs)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 342, in execute
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123]     cmd=sanitized_cmd)
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] ProcessExecutionError: Unexpected error while running command.
2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] COMMAND: SCP /VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123_RESIZE/DISK 192.0.2.21:/VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123/DISK
2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] EXIT CODE: 1
2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] STDOUT: U''
2017-07-11 14:10:15.555 27294 ERROR NOVA.COMPUTE.MANAGER [INSTANCE: ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123] STDERR: U'FORBIDDEN\NLOST CONNECTION\N'
2017-07-11 14:10:15.555 27294 ERROR nova.compute.manager [instance: ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123] 
2017-07-11 14:10:15.807 27294 ERROR oslo_messaging.rpc.dispatcher Command: scp /var/lib/nova/instances/ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123_resize/disk 192.0.2.21:/var/lib/nova/instances/ecdd4f8b-c0e5-48b1-ad56-4f1a426a7123/disk
+++

The error was basically the same, scp failure (line in capital letters here as well)

I performed a few more checks and found that nova user was not able to ssh into the other compute node as nova user even after configuring password less access by following the steps at [2]:

+++
bash-4.2$ ssh 192.0.2.19
This command must be run via SSH ForceCommand (see man 5 sshd_config).
Connection to 192.0.2.19 closed.

-bash-4.2$ ssh 192.0.2.21
This command must be run via SSH ForceCommand (see man 5 sshd_config).
Connection to 192.0.2.21 closed.
+++


After a little more digging, I found that the file "/var/lib/nova/.ssh/config" is coming from the openstack-nova-migration on the compute nodes:

[root@overcloud-compute-1 ~]# rpm -ql openstack-nova-migration-12.0.6-13.el7ost
/etc/nova/migration
/etc/nova/migration/authorized_keys
/etc/nova/migration/identity
/etc/nova/migration/rootwrap.conf
/etc/nova/migration/rootwrap.d
/etc/nova/migration/rootwrap.d/cold_migration.filters
/etc/sudoers.d/nova_migration
/usr/bin/nova-migration-wrapper
/var/lib/nova/.ssh
/var/lib/nova/.ssh/config
+++

Now by default, this is what is present in the "/var/lib/nova/.ssh/config" file:

+++
bash-4.2$ cat .ssh/config 
Host *
    User nova_migration
    UserKnownHostsFile /dev/null
    IdentityFile /etc/nova/migration/identity
+++

Now once I commented out these lines in the "/var/lib/nova/.ssh/config" file on both the compute nodes as below:

+++
-bash-4.2$ cat .ssh/config 
Host *
#    User nova_migration
#    UserKnownHostsFile /dev/null
#    IdentityFile /etc/nova/migration/identity
+++

I found that nova user was able to ssh into the other compute node as nova user
+++
-bash-4.2$ whoami 
nova

-bash-4.2$ ssh 192.0.2.19
Last login: Tue Jul 11 14:15:44 2017 from overcloud-compute-0.localdomain
-bash-4.2$ hostname
overcloud-compute-1.localdomain

-bash-4.2$ whoami
nova
+++

Once this was done, I launched another instance and tried to resize it again and it worked fine this time:

+++
[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks           |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | ACTIVE | -          | Running     | net1=172.168.122.5 |
+--------------------------------------+-------+--------+------------+-------------+--------------------+

[stack@instack ~(overcloud_admin)]$ nova show test3
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCFiskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | overcloud-compute-1.localdomain                          |  < = initially launched on compute-1
| OS-EXT-SRV-ATTR:hostname             | test3                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-compute-1.localdomain                          |



[stack@instack ~(overcloud_admin)]$ openstack server resize --flavor m1.custom test3
[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+------------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State       | Power State | Networks           |
+--------------------------------------+-------+--------+------------------+-------------+--------------------+
| 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | RESIZE | resize_migrating | Running     | net1=172.168.122.5 |
+--------------------------------------+-------+--------+------------------+-------------+--------------------+
[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+---------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State    | Power State | Networks           |
+--------------------------------------+-------+--------+---------------+-------------+--------------------+
| 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | RESIZE | resize_finish | Running     | net1=172.168.122.5 |
+--------------------------------------+-------+--------+---------------+-------------+--------------------+
[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+---------------+------------+-------------+--------------------+
| ID                                   | Name  | Status        | Task State | Power State | Networks           |
+--------------------------------------+-------+---------------+------------+-------------+--------------------+
| 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | VERIFY_RESIZE | -          | Running     | net1=172.168.122.5 |
+--------------------------------------+-------+---------------+------------+-------------+--------------------+

[stack@instack ~(overcloud_admin)]$ openstack server resize --confirm 74b33950-3dcd-4444-ba0e-20b08be35b09
[stack@instack ~(overcloud_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks           |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| 74b33950-3dcd-4444-ba0e-20b08be35b09 | test3 | ACTIVE | -          | Running     | net1=172.168.122.5 |  < = resized and disk moved to compute-0 
+--------------------------------------+-------+--------+------------+-------------+--------------------+

[stack@instack ~(overcloud_admin)]$ nova show test3
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCFiskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | overcloud-compute-0.localdomain                          |
| OS-EXT-SRV-ATTR:hostname             | test3                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-compute-0.localdomain                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000003                                        |


[root@overcloud-compute-0 ~]# virsh list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000003              running


Surprisingly enough, it works when directly used with virsh commands:

+++
[root@overcloud-compute-1 ~]#  virsh -c qemu+ssh://nova_migration@overcloud-compute-0/system?keyfile=/etc/nova/migration/identity 
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000003              running

virsh # 
+++


P.S I even tried setting "allow_resize_to_same_host" to "true" in nova.conf on the compute nodes, that failed too. It was only that after following the steps at [2] and then commenting out the above mentioned lines in "/var/lib/nova/.ssh/config" I was able to resize my instances.

[1] https://access.redhat.com/downloads/content/openstack-nova-migration/2015.1.4-36.el7ost/noarch/fd431d51/package

[2] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html-single/migrating_instances/#configure_ssh_tunneling_between_nodes

Additional info:

If the openstack-nova-migration package is taking care of
Comment 2 cshastri 2017-07-19 09:14:56 EDT
I just checked resizing an instance on OSP10 setup without setting up ssh tunneling, and the resize works without any issue.
Comment 3 Matthew Booth 2017-07-21 07:38:49 EDT
The issue here is that we're not whitelisting the correct scp command for OSP8. We can see from the logs (if you squint through the capitalisation) that the executed command is:

SCP /VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123_RESIZE/DISK 192.0.2.21:/VAR/LIB/NOVA/INSTANCES/ECDD4F8B-C0E5-48B1-AD56-4F1A426A7123/DISK

The nova_migration user is configured to pass ssh commands through nova-migration-wrapper, installed by openstack-nova-migration. This whitelists:

['scp', '-r', '-t']

This fails because the '-r', and '-t' options are missing, and the wrapper responds 'Forbidden'.

The -r argument has been present since version 14 (Newton, OSP 10). I can't see that the -t argument has ever been present, so either I'm looking in the wrong place or this has never worked for any version of OSP.

We need to fix the whitelisting in nova-migration-wrapper, and update it appropriately for all OSP versions.
Comment 4 Ollie Walsh 2017-07-21 10:57:31 EDT
Ignore the -t option, it's the hidden option that scp appends on the target side. i.e to allow scp <foo> <bar> we need to whitelist scp -t .... on the migration target host.

Note You need to log in before you can comment on or make changes to this bug.