Bug 1569039 - Live Migration fails
Summary: Live Migration fails
Keywords:
Status: NEW
Alias: None
Product: RDO
Classification: Community
Component: openstack-nova
Version: Pike
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: trunk
Assignee: nova-maint
QA Contact: nova-maint
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-18 14:24 UTC by Ricardo Noriega
Modified: 2018-04-18 14:26 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)
target-compute (513.48 KB, text/plain)
2018-04-18 14:24 UTC, Ricardo Noriega
no flags Details
source-compute (659.98 KB, text/plain)
2018-04-18 14:25 UTC, Ricardo Noriega
no flags Details
neutron-server (65.08 KB, text/plain)
2018-04-18 14:25 UTC, Ricardo Noriega
no flags Details

Description Ricardo Noriega 2018-04-18 14:24:20 UTC
Created attachment 1423619 [details]
target-compute

Description of problem:

Executing the following command:

openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2

It will show the VM in MIGRATING state for a while, and then it turns into ACTIVE, but still at the source host.


Version-Release number of selected component (if applicable):

[root@overcloud-novacompute-0 ~]# rpm -qa | grep nova
openstack-nova-console-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
puppet-nova-11.5.1-0.20180323030349.aae889c.el7.centos.noarch
openstack-nova-conductor-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-novncproxy-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
python2-novaclient-9.1.1-0.20171006101317.a1c0074.el7.centos.noarch
python-nova-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-migration-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-placement-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-scheduler-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-common-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-compute-16.1.1-0.20180322180942.69f996c.el7.centos.noarch


How reproducible:


Steps to Reproduce:
1. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2                                                                                                                                         
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000021                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2018-04-12T15:23:04.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | internal=172.20.1.14                                     |
| config_drive                        |                                                          |
| created                             | 2018-04-12T15:19:07Z                                     |
| flavor                              | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a)           |
| hostId                              | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 |
| id                                  | 30d280a2-be18-4e10-9cf5-a639de28034e                     |
| image                               | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8)            |
| key_name                            | undercloud-key                                           |
| name                                | test-cloud2                                              |
| progress                            | 0                                                        |
| project_id                          | 7ea4fe74f0944e79bf03114159b029d2                         |
| properties                          |                                                          |
| security_groups                     | name='allowall'                                          |
| status                              | ACTIVE                                                   |
| updated                             | 2018-04-18T11:16:09Z                                     |
| user_id                             | 2a989fcf2acc486e8b91be7e522bffec                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+


2.openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2

3. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2                                                                                                                                         
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000021                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2018-04-12T15:23:04.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | internal=172.20.1.14                                     |
| config_drive                        |                                                          |
| created                             | 2018-04-12T15:19:07Z                                     |
| flavor                              | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a)           |
| hostId                              | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 |
| id                                  | 30d280a2-be18-4e10-9cf5-a639de28034e                     |
| image                               | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8)            |
| key_name                            | undercloud-key                                           |
| name                                | test-cloud2                                              |
| progress                            | 0                                                        |
| project_id                          | 7ea4fe74f0944e79bf03114159b029d2                         |
| properties                          |                                                          |
| security_groups                     | name='allowall'                                          |
| status                              | ACTIVE                                                   |
| updated                             | 2018-04-18T11:16:09Z                                     |
| user_id                             | 2a989fcf2acc486e8b91be7e522bffec                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+


Actual results:

VM stays in the same source host.


Expected results:

VM would be allocated in the target host.


Additional info:

I can see the following error in the source host:

2018-04-18 11:07:42.466 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration@overcloud-novacompute-0.opnfvlf.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer: libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration@overcloud-novacompute-0.opnfvlf.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer
2018-04-18 11:07:42.466 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation thread notification thread_finished /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6779
2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.migration [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] VM running on src, migration failed find_job_type /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:232
2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6596
2018-04-18 11:07:42.791 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation has aborted


Check attachments.

Comment 1 Ricardo Noriega 2018-04-18 14:25:07 UTC
Created attachment 1423620 [details]
source-compute

Comment 2 Ricardo Noriega 2018-04-18 14:25:30 UTC
Created attachment 1423621 [details]
neutron-server


Note You need to log in before you can comment on or make changes to this bug.