Bug 1724131 - [OSP15] live migration aborts with Ncat: No such file or directory.: Input/output error
Summary: [OSP15] live migration aborts with Ncat: No such file or directory.: Input/ou...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: beta
: 15.0 (Stein)
Assignee: Martin Schuppert
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-26 10:16 UTC by Martin Schuppert
Modified: 2019-09-26 10:52 UTC (History)
4 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.5.1-0.20190701110422.889d4d4.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:34 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1834330 0 None None None 2019-06-26 12:53:47 UTC
OpenStack gerrit 667605 0 'None' MERGED Add /run/libvirt to nova_migration_target container 2020-08-24 04:28:13 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:23:51 UTC

Description Martin Schuppert 2019-06-26 10:16:23 UTC 
Description of problem:

performing live migration after make sure we have live_migration_wait_for_vif_plug=False in nova.conf on the computes.
Migration aborts with:

2019-06-26 08:17:43.518 6 INFO nova.compute.manager [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Took 6.75 seconds for pre_live_migration on destination host compute-0.localdomain.
2019-06-26 08:17:43.519 6 DEBUG nova.compute.manager [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Not waiting for events after pre_live_migration: [('network-vif-plugged', 'fbb880ff-b834-4cca-b12f-77a60a1984b2')].  _do_live_migration /usr/lib/python3.6/site-packages/nova/compute/manager.py:6548
...
2019-06-26 08:17:43.728 6 INFO nova.virt.libvirt.migration [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Increasing downtime to 50 ms after 0 sec elapsed time
2019-06-26 08:17:43.857 6 INFO nova.virt.libvirt.driver [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2019-06-26 08:17:44.293 6 ERROR nova.virt.libvirt.driver [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.localdomain:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Ncat: No such file or directory.: Input/output error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.localdomain:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Ncat: No such file or directory.: Input/output error
2019-06-26 08:17:44.295 6 DEBUG nova.virt.libvirt.driver [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Migration operation thread notification thread_finished /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:8015
2019-06-26 08:17:44.360 6 DEBUG nova.virt.libvirt.migration [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] VM running on src, migration failed find_job_type /usr/lib/python3.6/site-packages/nova/virt/libvirt/migration.py:360
2019-06-26 08:17:44.360 6 DEBUG nova.virt.libvirt.driver [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:7844
2019-06-26 08:17:44.361 6 ERROR nova.virt.libvirt.driver [-] [instance: a20ab13e-b348-4dc2-b294-e23865e2fc4b] Migration operation has aborted


Version-Release number of selected component (if applicable):
RHOS_TRUNK-15.0-RHEL-8-20190619.n.1

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
migration aborts and instance stays on src compute

Expected results:
migration works

Additional info:

during the migration, the disk files get copied to the new compute, so it is not a general connection issue between the computes:

[root@compute-0 ~]# ll -R /var/lib/nova/instances/a20ab13e-b348-4dc2-b294-e23865e2fc4b
/var/lib/nova/instances/a20ab13e-b348-4dc2-b294-e23865e2fc4b:
total 396
-rw-r--r--. 1 42436 42436 196616 Jun 26 08:47 disk
-rw-r--r--. 1 42436 42436 196624 Jun 26 08:47 disk.eph0
-rw-r--r--. 1 42436 42436    161 Jun 26 08:47 disk.info
Comment 1 Martin Schuppert 2019-06-26 12:53:05 UTC 
The issue was introduced with [1]. The nova-migration-wrapper inside the container needs access to the libvirt socket [1].

[1] https://review.opendev.org/#/c/662109/
[2] https://github.com/rdo-packages/nova-distgit/blob/rpm-master/nova-migration-wrapper#L31
Comment 2 Archit Modi 2019-06-26 15:12:40 UTC 
Facing the same issue with block migration:
(overcloud) [stack@undercloud-0 ~]$ nova --debug live-migration --block-migrate test-10267 compute-1.localdomain
DEBUG (extension:189) found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')
DEBUG (extension:189) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:189) found extension EntryPoint.parse('noauth = cinderclient.contrib.noauth:CinderNoAuthLoader')
DEBUG (extension:189) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:189) found extension EntryPoint.parse('none = keystoneauth1.loading._plugins.noauth:NoAuth')
DEBUG (extension:189) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:189) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:189) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:189) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:189) found extension EntryPoint.parse('v3adfspassword = keystoneauth1.extras._saml2._loading:ADFSPassword')
DEBUG (extension:189) found extension EntryPoint.parse('v3applicationcredential = keystoneauth1.loading._plugins.identity.v3:ApplicationCredential')
DEBUG (extension:189) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:189) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:189) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:189) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:189) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:189) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:189) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:189) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:189) found extension EntryPoint.parse('gnocchi-basic = gnocchiclient.auth:GnocchiBasicLoader')
DEBUG (extension:189) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.auth:GnocchiNoAuthLoader')
DEBUG (extension:189) found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader')
DEBUG (session:464) REQ: curl -g -i -X GET http://10.0.0.106:5000 -H "Accept: application/json" -H "User-Agent: nova keystoneauth1/3.13.1 python-requests/2.20.0 CPython/3.6.8"
DEBUG (connectionpool:205) Starting new HTTP connection (1): 10.0.0.106:5000
DEBUG (connectionpool:393) http://10.0.0.106:5000 "GET / HTTP/1.1" 300 265
DEBUG (session:495) RESP: [300] Content-Length: 265 Content-Type: application/json Date: Wed, 26 Jun 2019 15:01:05 GMT Location: http://10.0.0.106:5000/v3/ Server: Apache Vary: X-Auth-Token x-openstack-request-id: req-657151b2-ca7b-4244-91b4-d6f8cd2dd355
DEBUG (session:527) RESP BODY: {"versions": {"values": [{"id": "v3.12", "status": "stable", "updated": "2019-01-22T00:00:00Z", "links": [{"rel": "self", "href": "http://10.0.0.106:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
DEBUG (session:874) GET call to http://10.0.0.106:5000/ used request id req-657151b2-ca7b-4244-91b4-d6f8cd2dd355
DEBUG (base:176) Making authentication request to http://10.0.0.106:5000/v3/auth/tokens
DEBUG (connectionpool:393) http://10.0.0.106:5000 "POST /v3/auth/tokens HTTP/1.1" 201 8438
DEBUG (base:181) {"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "32e7226337394c4d8773c3187df8ccf6", "name": "admin", "password_expires_at": null}, "audit_ids": ["H3VbIkvlRQmlH8Zd1s_AxQ"], "expires_at": "2019-06-26T16:01:05.000000Z", "issued_at": "2019-06-26T15:01:05.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "b34c5eafd07d4fb1a45054dd72d404ae", "name": "admin"}, "is_domain": false, "roles": [{"id": "31c509bb587446a8ad462333e8e75115", "name": "_member_"}, {"id": "38734d32e44540cea802a348df01c8db", "name": "member"}, {"id": "a6c44f8833434c15ac63858e7464d713", "name": "admin"}, {"id": "35d100333bd849cd91706ac18fdd8499", "name": "reader"}], "catalog": [{"endpoints": [{"id": "04a8e5a3f9ff4954a6e87716f9595240", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8776/v2/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "7a29c0f2be1749a08b4ebe78e5fb07d8", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8776/v2/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "b24a2bc2ae104deeaf6e28f1dd19af70", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8776/v2/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}], "id": "0dd1022526d040cf87e060419964e421", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [], "id": "0e2cfa7561814ca5853170a8c5c86498", "type": "metering", "name": "ceilometer"}, {"endpoints": [{"id": "15c65f44f547467186c5f152d9a67e49", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8041", "region": "regionOne"}, {"id": "adc65be7b854428da294bb935d674bf5", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8041", "region": "regionOne"}, {"id": "fd8baa7969144b5c8391fb8f2a8e6826", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8041", "region": "regionOne"}], "id": "23f9918f85454c0f89bff8feca7518d6", "type": "metric", "name": "gnocchi"}, {"endpoints": [{"id": "4629bdac23114fdb9222238124148465", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8776/v3/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "87a45c13e4894541b8d72fde7f9256e4", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8776/v3/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "de29a140e1a2483ea95c246c8464366e", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8776/v3/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}], "id": "2b8158ac1df7435d9ec7db8295406bd0", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "0b2a5323c44d4078bb4281c0778a0cd3", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:9696", "region": "regionOne"}, {"id": "1c1c046b27b2476e98c0596c3abeb344", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:9696", "region": "regionOne"}, {"id": "b7a92f9cfa184258a7992de607c409bd", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:9696", "region": "regionOne"}], "id": "531ce2310eb54db1bce70b215884b3d5", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0425ed8f53164d52a06f758d2a8b252e", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8774/v2.1", "region": "regionOne"}, {"id": "57dcc1a1bdda4f2d960a3adb05e62d24", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8774/v2.1", "region": "regionOne"}, {"id": "57f2bfa1344144f3b8c998a674fb5c39", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8774/v2.1", "region": "regionOne"}], "id": "61969c579df54955bfffbb912e967933", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "39caad3b1879416e9f9ae238d5c7b77b", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8080/v1/AUTH_b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "47b77ae2619d4813b87dc7e83719ce32", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.3.136:8080", "region": "regionOne"}, {"id": "7beca9f71e3f4e6eb4de79725d6ed0ce", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.3.136:8080/v1/AUTH_b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}], "id": "7ce3dec712a848d1940f9b3d3193e251", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "114a0640d0dd46138e1adb0f4e97c4ba", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8042", "region": "regionOne"}, {"id": "bd2f32b7f03c480eb7d9c9d21921b2f3", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8042", "region": "regionOne"}, {"id": "c322ccb81cee4aa69539df4ff794f753", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8042", "region": "regionOne"}], "id": "811c5bc917f741fe97aabca115ef05e9", "type": "alarming", "name": "aodh"}, {"endpoints": [{"id": "419e4b31757e4321ab250117b8f7953b", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8778/placement", "region": "regionOne"}, {"id": "71cbf54faea14d04b029dd25fa525ba9", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8778/placement", "region": "regionOne"}, {"id": "bcfa20f8d77d419cafe9d9ca4783fc9d", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8778/placement", "region": "regionOne"}], "id": "98b66062855e4f9ebc49e9c2de2198c0", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "18c8c6a01c6d4376920c6163d926761a", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.21:35357", "region": "regionOne"}, {"id": "a2a37b4973434c69bf9325c84470caa1", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:5000", "region": "regionOne"}, {"id": "f110f3cd583c4b17b014b86f2b597e83", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:5000", "region": "regionOne"}], "id": "b1fef497ef2d4803a8236626251601b1", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "5c5aca494f2b486795358e4a54ef7f5f", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:9292", "region": "regionOne"}, {"id": "ae42136e45ea44fe974d485a8b69b378", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:9292", "region": "regionOne"}, {"id": "b17dc68f853440ebb5a4cc3d86c08ada", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:9292", "region": "regionOne"}], "id": "b72f10229b2e458783ca32755a35d97e", "type": "image", "name": "glance"}, {"endpoints": [{"id": "623f0c1bbfdb43049b9548ee8841532c", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8004/v1/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "a10d5e8dbeb84408aa616ca41d884d89", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8004/v1/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}, {"id": "dfc23905be2d45baa4b8ac2268926995", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8004/v1/b34c5eafd07d4fb1a45054dd72d404ae", "region": "regionOne"}], "id": "d06fc380aacf41fda1dca4e78c72e79c", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "10cefcbc1f284234a7e5a2619f7ba8ea", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8977", "region": "regionOne"}, {"id": "80ddfb61a3684789b10c4cdcc45b7036", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8977", "region": "regionOne"}, {"id": "fd44fd5c033345dcbba7565eca1a5e30", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8977", "region": "regionOne"}], "id": "de6dc3b5713e4e25af5089e6c3e1e394", "type": "event", "name": "panko"}, {"endpoints": [{"id": "1d7fb2154f7c40fba5d08169cfac0de1", "interface": "public", "region_id": "regionOne", "url": "http://10.0.0.106:8000/v1", "region": "regionOne"}, {"id": "2a842473eeef43f987c999e0f4f1662a", "interface": "admin", "region_id": "regionOne", "url": "http://172.17.1.108:8000/v1", "region": "regionOne"}, {"id": "f91194f1b68344859e548fb6785c1f65", "interface": "internal", "region_id": "regionOne", "url": "http://172.17.1.108:8000/v1", "region": "regionOne"}], "id": "f27ef3cf2f614b2ea0d2c4813cb1f520", "type": "cloudformation", "name": "heat-cfn"}]}}
REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1 -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}b9d45020a7d178bfe2eee3bd84f44d904e52ca37d203b7393cb2979b27615bce"
DEBUG (session:464) REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1 -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}b9d45020a7d178bfe2eee3bd84f44d904e52ca37d203b7393cb2979b27615bce"
DEBUG (connectionpool:205) Starting new HTTP connection (1): 10.0.0.106:8774
DEBUG (connectionpool:393) http://10.0.0.106:8774 "GET /v2.1 HTTP/1.1" 302 None
RESP: [302] Content-Type: text/plain; charset=utf8 Date: Wed, 26 Jun 2019 15:01:05 GMT Location: http://10.0.0.106:8774/v2.1/ Server: Apache Transfer-Encoding: chunked x-compute-request-id: req-da2b0609-a716-4da2-80d9-c4ad4e3af015 x-openstack-request-id: req-da2b0609-a716-4da2-80d9-c4ad4e3af015
DEBUG (session:495) RESP: [302] Content-Type: text/plain; charset=utf8 Date: Wed, 26 Jun 2019 15:01:05 GMT Location: http://10.0.0.106:8774/v2.1/ Server: Apache Transfer-Encoding: chunked x-compute-request-id: req-da2b0609-a716-4da2-80d9-c4ad4e3af015 x-openstack-request-id: req-da2b0609-a716-4da2-80d9-c4ad4e3af015
RESP BODY: Omitted, Content-Type is set to text/plain; charset=utf8. Only application/json responses have their bodies logged.
DEBUG (session:527) RESP BODY: Omitted, Content-Type is set to text/plain; charset=utf8. Only application/json responses have their bodies logged.
DEBUG (connectionpool:393) http://10.0.0.106:8774 "GET /v2.1/ HTTP/1.1" 200 None
RESP: [200] Content-Encoding: gzip Content-Type: application/json Date: Wed, 26 Jun 2019 15:01:06 GMT OpenStack-API-Version: compute 2.1 Server: Apache Transfer-Encoding: chunked Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-19cec0db-af87-48a3-99d5-588d49ab4a06 x-openstack-request-id: req-19cec0db-af87-48a3-99d5-588d49ab4a06
DEBUG (session:495) RESP: [200] Content-Encoding: gzip Content-Type: application/json Date: Wed, 26 Jun 2019 15:01:06 GMT OpenStack-API-Version: compute 2.1 Server: Apache Transfer-Encoding: chunked Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-19cec0db-af87-48a3-99d5-588d49ab4a06 x-openstack-request-id: req-19cec0db-af87-48a3-99d5-588d49ab4a06
RESP BODY: {"version": {"id": "v2.1", "status": "CURRENT", "version": "2.72", "min_version": "2.1", "updated": "2013-07-23T11:33:21Z", "links": [{"rel": "self", "href": "http://10.0.0.106:8774/v2.1/"}, {"rel": "describedby", "type": "text/html", "href": "http://docs.openstack.org/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}]}}
DEBUG (session:527) RESP BODY: {"version": {"id": "v2.1", "status": "CURRENT", "version": "2.72", "min_version": "2.1", "updated": "2013-07-23T11:33:21Z", "links": [{"rel": "self", "href": "http://10.0.0.106:8774/v2.1/"}, {"rel": "describedby", "type": "text/html", "href": "http://docs.openstack.org/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}]}}
GET call to compute for http://10.0.0.106:8774/v2.1/ used request id req-19cec0db-af87-48a3-99d5-588d49ab4a06
DEBUG (session:865) GET call to compute for http://10.0.0.106:8774/v2.1/ used request id req-19cec0db-af87-48a3-99d5-588d49ab4a06
DEBUG (extension:189) found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')
DEBUG (extension:189) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:189) found extension EntryPoint.parse('noauth = cinderclient.contrib.noauth:CinderNoAuthLoader')
DEBUG (extension:189) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:189) found extension EntryPoint.parse('none = keystoneauth1.loading._plugins.noauth:NoAuth')
DEBUG (extension:189) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:189) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:189) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:189) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:189) found extension EntryPoint.parse('v3adfspassword = keystoneauth1.extras._saml2._loading:ADFSPassword')
DEBUG (extension:189) found extension EntryPoint.parse('v3applicationcredential = keystoneauth1.loading._plugins.identity.v3:ApplicationCredential')
DEBUG (extension:189) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:189) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:189) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:189) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:189) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:189) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:189) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:189) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:189) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:189) found extension EntryPoint.parse('gnocchi-basic = gnocchiclient.auth:GnocchiBasicLoader')
DEBUG (extension:189) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.auth:GnocchiNoAuthLoader')
DEBUG (extension:189) found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader')
DEBUG (session:464) REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1/servers?name=test-10267 -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.72" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}b9d45020a7d178bfe2eee3bd84f44d904e52ca37d203b7393cb2979b27615bce" -H "X-OpenStack-Nova-API-Version: 2.72"
DEBUG (connectionpool:393) http://10.0.0.106:8774 "GET /v2.1/servers?name=test-10267 HTTP/1.1" 200 None
DEBUG (session:495) RESP: [200] Content-Encoding: gzip Content-Type: application/json Date: Wed, 26 Jun 2019 15:01:06 GMT OpenStack-API-Version: compute 2.72 Server: Apache Transfer-Encoding: chunked Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding X-OpenStack-Nova-API-Version: 2.72 x-compute-request-id: req-8056d159-bf9b-4858-b898-e7d9b763283a x-openstack-request-id: req-8056d159-bf9b-4858-b898-e7d9b763283a
DEBUG (session:527) RESP BODY: {"servers": [{"id": "0cab7b52-f218-4750-a164-6cf1d83f1ef3", "name": "test-10267", "links": [{"rel": "self", "href": "http://10.0.0.106:8774/v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3"}, {"rel": "bookmark", "href": "http://10.0.0.106:8774/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3"}]}]}
DEBUG (session:865) GET call to compute for http://10.0.0.106:8774/v2.1/servers?name=test-10267 used request id req-8056d159-bf9b-4858-b898-e7d9b763283a
DEBUG (session:464) REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3 -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.72" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}b9d45020a7d178bfe2eee3bd84f44d904e52ca37d203b7393cb2979b27615bce" -H "X-OpenStack-Nova-API-Version: 2.72"
DEBUG (connectionpool:393) http://10.0.0.106:8774 "GET /v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3 HTTP/1.1" 200 None
DEBUG (session:495) RESP: [200] Content-Encoding: gzip Content-Type: application/json Date: Wed, 26 Jun 2019 15:01:06 GMT OpenStack-API-Version: compute 2.72 Server: Apache Transfer-Encoding: chunked Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding X-OpenStack-Nova-API-Version: 2.72 x-compute-request-id: req-b95aff76-fc48-4151-a5fa-d2ca2a02b718 x-openstack-request-id: req-b95aff76-fc48-4151-a5fa-d2ca2a02b718
DEBUG (session:527) RESP BODY: {"server": {"id": "0cab7b52-f218-4750-a164-6cf1d83f1ef3", "name": "test-10267", "status": "ACTIVE", "tenant_id": "b34c5eafd07d4fb1a45054dd72d404ae", "user_id": "32e7226337394c4d8773c3187df8ccf6", "metadata": {}, "hostId": "5bc7e05bbf53151752e60b3e33cf3b675e56674fe3ba7b144f61bec4", "image": {"id": "1b89ad70-cb5b-4b22-91d5-ab3e49bbdf18", "links": [{"rel": "bookmark", "href": "http://10.0.0.106:8774/images/1b89ad70-cb5b-4b22-91d5-ab3e49bbdf18"}]}, "flavor": {"vcpus": 1, "ram": 512, "disk": 11, "ephemeral": 0, "swap": 0, "original_name": "m1.custom", "extra_specs": {}}, "created": "2019-06-26T13:59:44Z", "updated": "2019-06-26T13:59:57Z", "addresses": {"private": [{"version": 4, "addr": "192.168.100.85", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:09:49:4c"}, {"version": 4, "addr": "10.0.0.249", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:09:49:4c"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.106:8774/v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3"}, {"rel": "bookmark", "href": "http://10.0.0.106:8774/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-26T13:59:57.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-SRV-ATTR:host": "compute-0.localdomain", "OS-EXT-SRV-ATTR:instance_name": "instance-0000009c", "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute-0.localdomain", "OS-EXT-SRV-ATTR:reservation_id": "r-bmg4e00k", "OS-EXT-SRV-ATTR:launch_index": 0, "OS-EXT-SRV-ATTR:hostname": "test-10267", "OS-EXT-SRV-ATTR:kernel_id": "", "OS-EXT-SRV-ATTR:ramdisk_id": "", "OS-EXT-SRV-ATTR:root_device_name": "/dev/vda", "OS-EXT-SRV-ATTR:user_data": null, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "host_status": "UP", "locked": false, "description": "test-10267", "tags": [], "trusted_image_certificates": null, "server_groups": []}}
DEBUG (session:865) GET call to compute for http://10.0.0.106:8774/v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3 used request id req-b95aff76-fc48-4151-a5fa-d2ca2a02b718
DEBUG (session:464) REQ: curl -g -i -X POST http://10.0.0.106:8774/v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3/action -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: compute 2.72" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}b9d45020a7d178bfe2eee3bd84f44d904e52ca37d203b7393cb2979b27615bce" -H "X-OpenStack-Nova-API-Version: 2.72" -d '{"os-migrateLive": {"host": "compute-1.localdomain", "block_migration": true}}'
DEBUG (connectionpool:393) http://10.0.0.106:8774 "POST /v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3/action HTTP/1.1" 202 0
DEBUG (session:495) RESP: [202] Content-Length: 0 Content-Type: application/json Date: Wed, 26 Jun 2019 15:01:07 GMT OpenStack-API-Version: compute 2.72 Server: Apache Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.72 x-compute-request-id: req-67b11130-921e-453d-9cdf-df7c636908c7 x-openstack-request-id: req-67b11130-921e-453d-9cdf-df7c636908c7
DEBUG (session:865) POST call to compute for http://10.0.0.106:8774/v2.1/servers/0cab7b52-f218-4750-a164-6cf1d83f1ef3/action used request id req-67b11130-921e-453d-9cdf-df7c636908c7

Logs:
2019-06-26 15:01:19.312 9 ERROR nova.virt.libvirt.driver [-] [instance: 0cab7b52-f218-4750-a164-6cf1d83f1ef3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.localdomain:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Ncat: No such file or directory.: Input/output error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova_migration.localdomain:2022/system?keyfile=/etc/nova/migration/identity: End of file while reading data: Ncat: No such file or directory.: Input/output error
2019-06-26 15:01:19.850 9 ERROR nova.virt.libvirt.driver [-] [instance: 0cab7b52-f218-4750-a164-6cf1d83f1ef3] Migration operation has aborted
Comment 7 Joe H. Rahme 2019-07-10 12:06:05 UTC 
Live Migration works now on osp15
Comment 11 errata-xmlrpc 2019-09-21 11:23:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811
Note You need to log in before you can comment on or make changes to this bug.