Bug 1569039
| Summary: | Live Migration fails | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Community] RDO | Reporter: | Ricardo Noriega <rnoriega> | ||||||||
| Component: | openstack-nova | Assignee: | OSP DFG:Compute <osp-dfg-compute> | ||||||||
| Status: | NEW --- | QA Contact: | OSP DFG:Compute <osp-dfg-compute> | ||||||||
| Severity: | unspecified | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | Pike | ||||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | trunk | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | Type: | Bug | |||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Created attachment 1423620 [details]
source-compute
Created attachment 1423621 [details]
neutron-server
|
Created attachment 1423619 [details] target-compute Description of problem: Executing the following command: openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2 It will show the VM in MIGRATING state for a while, and then it turns into ACTIVE, but still at the source host. Version-Release number of selected component (if applicable): [root@overcloud-novacompute-0 ~]# rpm -qa | grep nova openstack-nova-console-16.1.1-0.20180322180942.69f996c.el7.centos.noarch puppet-nova-11.5.1-0.20180323030349.aae889c.el7.centos.noarch openstack-nova-conductor-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-novncproxy-16.1.1-0.20180322180942.69f996c.el7.centos.noarch python2-novaclient-9.1.1-0.20171006101317.a1c0074.el7.centos.noarch python-nova-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-migration-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-placement-api-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-scheduler-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-common-16.1.1-0.20180322180942.69f996c.el7.centos.noarch openstack-nova-compute-16.1.1-0.20180322180942.69f996c.el7.centos.noarch How reproducible: Steps to Reproduce: 1. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2 +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-novacompute-1.opnfvlf.org | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org | | OS-EXT-SRV-ATTR:instance_name | instance-00000021 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2018-04-12T15:23:04.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | internal=172.20.1.14 | | config_drive | | | created | 2018-04-12T15:19:07Z | | flavor | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a) | | hostId | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 | | id | 30d280a2-be18-4e10-9cf5-a639de28034e | | image | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8) | | key_name | undercloud-key | | name | test-cloud2 | | progress | 0 | | project_id | 7ea4fe74f0944e79bf03114159b029d2 | | properties | | | security_groups | name='allowall' | | status | ACTIVE | | updated | 2018-04-18T11:16:09Z | | user_id | 2a989fcf2acc486e8b91be7e522bffec | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+ 2.openstack server migrate --live overcloud-novacompute-0.opnfvlf.org test-cloud2 3. (overcloud) [stack@undercloud ~]$ openstack server show test-cloud2 +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-novacompute-1.opnfvlf.org | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.opnfvlf.org | | OS-EXT-SRV-ATTR:instance_name | instance-00000021 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2018-04-12T15:23:04.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | internal=172.20.1.14 | | config_drive | | | created | 2018-04-12T15:19:07Z | | flavor | m1.tiny (42abe377-4948-4605-81ed-a149d9b3e56a) | | hostId | f42da2e7acf5bcf6755b17c2fa311071c6c3add98a8a69984591a1c1 | | id | 30d280a2-be18-4e10-9cf5-a639de28034e | | image | cirros (efc10390-7b50-4a0f-bcac-303ef271d6f8) | | key_name | undercloud-key | | name | test-cloud2 | | progress | 0 | | project_id | 7ea4fe74f0944e79bf03114159b029d2 | | properties | | | security_groups | name='allowall' | | status | ACTIVE | | updated | 2018-04-18T11:16:09Z | | user_id | 2a989fcf2acc486e8b91be7e522bffec | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+ Actual results: VM stays in the same source host. Expected results: VM would be allocated in the target host. Additional info: I can see the following error in the source host: 2018-04-18 11:07:42.466 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer: libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.org:2022/system?keyfile=/etc/nova/migration/identity: Cannot recv data: ssh: connect to host overcloud-novacompute-0.opnfvlf.org port 2022: Connection timed out: Connection reset by peer 2018-04-18 11:07:42.466 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation thread notification thread_finished /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6779 2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.migration [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] VM running on src, migration failed find_job_type /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:232 2018-04-18 11:07:42.790 3031 DEBUG nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6596 2018-04-18 11:07:42.791 3031 ERROR nova.virt.libvirt.driver [req-5b81d8a0-ea3a-47da-b4a9-ee27544f6019 b4d6211d60f844b0a3f0d09837488139 62c3b59e249e44bca4da05ffdffe2943 - default default] [instance: 30d280a2-be18-4e10-9cf5-a639de28034e] Migration operation has aborted Check attachments.