Created attachment 1423619 [details]
target-compute

Description of problem:

Executing the following command:

openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2

It will show the VM in MIGRATING state for a while, and then it turns into ACTIVE, but still at the source host.


Version-Release number of selected component (if applicable):

[root@overcloud-novacompute-0 ~]# rpm -qa | grep nova
openstack-nova-console-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
puppet-nova-11.5.1-0.20180323030349.aae889c.el7.centos.noarch
openstack-nova-conductor-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-novncproxy-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
python2-novaclient-9.1.1-0.20171006101317.a1c0074.el7.centos.noarch
python-nova-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-migration-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-placement-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-scheduler-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-common-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-compute-16.1.1-0.20180322180942.69f996c.el7.centos.noarch


How reproducible:


Steps to Reproduce:
1. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2                                                                                                                                         
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000021                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2018-04-12T15:23:04.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | internal=172.20.1.14                                     |
| config_drive                        |                                                          |
| created                             | 2018-04-12T15:19:07Z                                     |
| flavor                              | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a)           |
| hostId                              | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 |
| id                                  | 30d280a2-be18-4e10-9cf5-a639de28034e                     |
| image                               | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8)            |
| key_name                            | undercloud-key                                           |
| name                                | test-cloud2                                              |
| progress                            | 0                                                        |
| project_id                          | 7ea4fe74f0944e79bf03114159b029d2                         |
| properties                          |                                                          |
| security_groups                     | name='allowall'                                          |
| status                              | ACTIVE                                                   |
| updated                             | 2018-04-18T11:16:09Z                                     |
| user_id                             | 2a989fcf2acc486e8b91be7e522bffec                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+


2.openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2

3. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2                                                                                                                                         
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000021                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2018-04-12T15:23:04.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | internal=172.20.1.14                                     |
| config_drive                        |                                                          |
| created                             | 2018-04-12T15:19:07Z                                     |
| flavor                              | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a)           |
| hostId                              | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 |
| id                                  | 30d280a2-be18-4e10-9cf5-a639de28034e                     |
| image                               | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8)            |
| key_name                            | undercloud-key                                           |
| name                                | test-cloud2                                              |
| progress                            | 0                                                        |
| project_id                          | 7ea4fe74f0944e79bf03114159b029d2                         |
| properties                          |                                                          |
| security_groups                     | name='allowall'                                          |
| status                              | ACTIVE                                                   |
| updated                             | 2018-04-18T11:16:09Z                                     |
| user_id                             | 2a989fcf2acc486e8b91be7e522bffec                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+


Actual results:

VM stays in the same source host.


Expected results:

VM would be allocated in the target host.


Additional info:

I can see the following error in the source host:

2018-04-18 11:07:42.466 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration@overcloud-novacompute-0.opnfvlf.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer: libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration@overcloud-novacompute-0.opnfvlf.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer
2018-04-18 11:07:42.466 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation thread notification thread_finished /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6779
2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.migration [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] VM running on src, migration failed find_job_type /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:232
2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6596
2018-04-18 11:07:42.791 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation has aborted


Check attachments.