RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/
Bug 1569039 - Live Migration fails
Summary: Live Migration fails
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: RDO
Classification: Community
Component: openstack-nova
Version: Pike
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: trunk
Assignee: OSP DFG:Compute
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-18 14:24 UTC by Ricardo Noriega
Modified: 2025-02-10 03:59 UTC (History)
0 users

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2025-02-10 03:59:15 UTC
Embargoed:


Attachments (Terms of Use)
target-compute (513.48 KB, text/plain)
2018-04-18 14:24 UTC, Ricardo Noriega
no flags Details
source-compute (659.98 KB, text/plain)
2018-04-18 14:25 UTC, Ricardo Noriega
no flags Details
neutron-server (65.08 KB, text/plain)
2018-04-18 14:25 UTC, Ricardo Noriega
no flags Details

Description Ricardo Noriega 2018-04-18 14:24:20 UTC
Created attachment 1423619 [details]
target-compute

Description of problem:

Executing the following command:

openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2

It will show the VM in MIGRATING state for a while, and then it turns into ACTIVE, but still at the source host.


Version-Release number of selected component (if applicable):

[root@overcloud-novacompute-0 ~]# rpm -qa | grep nova
openstack-nova-console-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
puppet-nova-11.5.1-0.20180323030349.aae889c.el7.centos.noarch
openstack-nova-conductor-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-novncproxy-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
python2-novaclient-9.1.1-0.20171006101317.a1c0074.el7.centos.noarch
python-nova-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-migration-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-placement-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-scheduler-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-common-16.1.1-0.20180322180942.69f996c.el7.centos.noarch
openstack-nova-compute-16.1.1-0.20180322180942.69f996c.el7.centos.noarch


How reproducible:


Steps to Reproduce:
1. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2                                                                                                                                         
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000021                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2018-04-12T15:23:04.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | internal=172.20.1.14                                     |
| config_drive                        |                                                          |
| created                             | 2018-04-12T15:19:07Z                                     |
| flavor                              | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a)           |
| hostId                              | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 |
| id                                  | 30d280a2-be18-4e10-9cf5-a639de28034e                     |
| image                               | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8)            |
| key_name                            | undercloud-key                                           |
| name                                | test-cloud2                                              |
| progress                            | 0                                                        |
| project_id                          | 7ea4fe74f0944e79bf03114159b029d2                         |
| properties                          |                                                          |
| security_groups                     | name='allowall'                                          |
| status                              | ACTIVE                                                   |
| updated                             | 2018-04-18T11:16:09Z                                     |
| user_id                             | 2a989fcf2acc486e8b91be7e522bffec                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+


2.openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2

3. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2                                                                                                                                         
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000021                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2018-04-12T15:23:04.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | internal=172.20.1.14                                     |
| config_drive                        |                                                          |
| created                             | 2018-04-12T15:19:07Z                                     |
| flavor                              | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a)           |
| hostId                              | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 |
| id                                  | 30d280a2-be18-4e10-9cf5-a639de28034e                     |
| image                               | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8)            |
| key_name                            | undercloud-key                                           |
| name                                | test-cloud2                                              |
| progress                            | 0                                                        |
| project_id                          | 7ea4fe74f0944e79bf03114159b029d2                         |
| properties                          |                                                          |
| security_groups                     | name='allowall'                                          |
| status                              | ACTIVE                                                   |
| updated                             | 2018-04-18T11:16:09Z                                     |
| user_id                             | 2a989fcf2acc486e8b91be7e522bffec                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+


Actual results:

VM stays in the same source host.


Expected results:

VM would be allocated in the target host.


Additional info:

I can see the following error in the source host:

2018-04-18 11:07:42.466 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer: libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer
2018-04-18 11:07:42.466 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation thread notification thread_finished /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6779
2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.migration [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] VM running on src, migration failed find_job_type /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:232
2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6596
2018-04-18 11:07:42.791 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation has aborted


Check attachments.

Comment 1 Ricardo Noriega 2018-04-18 14:25:07 UTC
Created attachment 1423620 [details]
source-compute

Comment 2 Ricardo Noriega 2018-04-18 14:25:30 UTC
Created attachment 1423621 [details]
neutron-server

Comment 4 Red Hat Bugzilla 2025-02-10 03:59:15 UTC
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.


Note You need to log in before you can comment on or make changes to this bug.