Description of problem: Deploy RHOS 14 with mentioned topology and NovaNFS backend. Run the following tempest tests which fails: 1. tempest.api.compute.admin.test_live_migration.LiveMigrationRemoteConsolesV26Test. test_live_block_migration[id-1dce86b8-eb04-4c03-a9d8-9c1dc3ee0c7b] test_live_block_migration_paused[id-1e107f21-61b2-4988-8f22-b196e938ab88] 2. tempest.api.compute.admin.test_live_migration.LiveMigrationTest. test_live_block_migration[id-1dce86b8-eb04-4c03-a9d8-9c1dc3ee0c7b] test_live_block_migration_paused[id-1e107f21-61b2-4988-8f22-b196e938ab88] Topology: 1 controller, 2 compute, NovaNFS backend Version-Release number of selected component (if applicable): core_puddle: 2019-05-31.1 Steps to Reproduce: 1. Deploy RHOS 14 with mentioned topology and NovaNFS backend. 2. Run the following tempest tests which fails: tempest.api.compute.admin.test_live_migration.LiveMigrationRemoteConsolesV26Test. test_live_block_migration[id-1dce86b8-eb04-4c03-a9d8-9c1dc3ee0c7b] test_live_block_migration_paused[id-1e107f21-61b2-4988-8f22-b196e938ab88] tempest.api.compute.admin.test_live_migration.LiveMigrationTest. test_live_block_migration[id-1dce86b8-eb04-4c03-a9d8-9c1dc3ee0c7b] test_live_block_migration_paused[id-1e107f21-61b2-4988-8f22-b196e938ab88] Actual results: 2019-06-03 18:59:47,468 396971 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.6', 'X-Auth-Token': '<omitted>'} Body: {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": "compute-0.localdomain"}} Response - Headers: {'status': '400', u'content-length': '193', 'content-location': 'http://10.0.0.110:8774/v2.1/servers/69b07a78-c952-4122-9fe5-02c4d18f2bc5/action', u'x-compute-request-id': 'req-0ae5612a-d5a1-4b42-a1db-ee19dd295608', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'server': 'Apache', u'openstack-api-version': 'compute 2.6', u'connection': 'close', u'x-openstack-nova-api-version': '2.6', u'date': 'Mon, 03 Jun 2019 22:59:43 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'req-0ae5612a-d5a1-4b42-a1db-ee19dd295608'} Body: {"badRequest": {"message": "compute-1.localdomain is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.", "code": 400}} }}} Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_live_migration.py", line 128, in test_live_block_migration self._test_live_migration() File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_live_migration.py", line 118, in _test_live_migration self._live_migrate(server_id, destination_host, state, volume_backed) File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_live_migration.py", line 80, in _live_migrate self._migrate_server_to(server_id, target_host, volume_backed) File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_live_migration.py", line 76, in _migrate_server_to **kwargs) File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 502, in live_migrate_server return self.action(server_id, 'os-migrateLive', **kwargs) File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 203, in action post_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 279, in post return self.request('POST', url, extra_headers, headers, body, chunked) File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request method, url, extra_headers, headers, body, chunked) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 670, in request self._error_checker(resp, resp_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 781, in _error_checker raise exceptions.BadRequest(resp_body, resp=resp) tempest.lib.exceptions.BadRequest: Bad request Details: {u'message': u'compute-1.localdomain is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.', u'code': 400} Expected results: Tests passed Additional info:
This is the code check for whether the exception will be raised: elif not (dest_check_data.is_shared_block_storage or dest_check_data.is_shared_instance_path): reason = _("Shared storage live-migration requires either shared " "storage or boot-from-volume with no local disks.") raise exception.InvalidSharedStorage(reason=reason, path=source) dest_check_data.is_shared_instance_path checks whether an expected tmpfile exists in the CONF.instances_path. If it does, it thinks shared storage = True. I found a log message (for a different instance than the one described in the bug report): 2019-06-03 23:00:12.551 1 DEBUG nova.virt.libvirt.driver [req-7642aef5-3e4b-49c5-a4ae-c363d5bb5067 1a4f907c19d942349868dea486687d6f ae5cd2406b4a469ea7ffcea0393b9d25 - default default] [instance: 579419b5-c0f0-4835-8a1a-c424ec54c6f4] Check if temp file /var/lib/nova/instances/tmpxkhHki exists to indicate shared storage is being used for migration. Exists? False _check_shared_storage_test_file /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6951 indicating that the check for the tmpfile failed and the code is determining that shared storage is not being used. There is a note in the code for Stein and Rocky stating that under heavy IO, the check can erroneously fail: https://github.com/openstack/nova/blob/stable/stein/nova/virt/libvirt/driver.py#L7485 https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L6936 So, I wonder if that is what's happening in our CI?
Looking at a different live migration: tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration with "block_migration=True,is_shared_block_storage=False,is_shared_instance_path=False", it passed: 2019-06-03 23:00:13.735 1 INFO nova.compute.manager [req-66ac6939-8b2d-4703-a624-b6fa84e9e536 e5215363908f40649a66a11c0ac5dea8 52b68a446d904fa5a2bcc912b29eaafd - default default] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Took 8.12 seconds to build instance. 2019-06-03 23:00:20.771 1 DEBUG nova.virt.libvirt.driver [req-75af898f-61b6-41ce-b5c2-ac2fe85ee879 278b2a1a4d7f44f4b24a11cff7dbbb6f d92796165d47400ba0c0454368376b6d - default default] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Check if temp file /var/lib/nova/instances/tmpdtCugf exists to indicate shared storage is being used for migration. Exists? False _check_shared_storage_test_file /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6951 2019-06-03 23:00:20.772 1 DEBUG nova.compute.manager [req-75af898f-61b6-41ce-b5c2-ac2fe85ee879 278b2a1a4d7f44f4b24a11cff7dbbb6f d92796165d47400ba0c0454368376b6d - default default] source check data is LibvirtLiveMigrateData(bdms=<?>,block_migration=True,disk_available_mb=44032,disk_over_commit=<?>,dst_wants_file_backed_memory=False,file_backed_memory_discard=False,filename='tmpdtCugf',graphics_listen_addr_spice=127.0.0.1,graphics_listen_addr_vnc=172.17.1.14,image_type='default',instance_relative_path='92c8e947-31bc-4dd3-aa6e-6666275d9dea',is_shared_block_storage=False,is_shared_instance_path=False,is_volume_backed=False,migration=<?>,old_vol_attachment_ids=<?>,serial_listen_addr=None,serial_listen_ports=<?>,src_supports_native_luks=True,supported_perf_events=<?>,target_connect_addr=<?>,vifs=<?>,wait_for_vif_plugged=<?>) check_can_live_migrate_source /usr/lib/python2.7/site-packages/nova/compute/manager.py:6152 2019-06-03 23:00:25.976 1 DEBUG nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Preparing to wait for external event network-vif-plugged-1273b802-b7a6-4af0-bb65-5a0e8f7f5902 prepare_for_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:328 2019-06-03 23:00:31.174 1 DEBUG nova.compute.manager [req-d0064a1e-ef81-4825-8287-05c64b6503f2 6d30bccbef8b4f098c16b1184e710e40 95a7af67e36e4605977726510f8c061f - default default] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Received event network-vif-unplugged-1273b802-b7a6-4af0-bb65-5a0e8f7f5902 external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:8217 2019-06-03 23:00:32.135 1 INFO nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Took 6.16 seconds for pre_live_migration on destination host compute-1.localdomain. 2019-06-03 23:00:32.136 1 DEBUG nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Not waiting for events after pre_live_migration: [('network-vif-plugged', u'1273b802-b7a6-4af0-bb65-5a0e8f7f5902')]. _do_live_migration /usr/lib/python2.7/site-packages/nova/compute/manager.py:6381 2019-06-03 23:00:32.164 1 DEBUG nova.compute.manager [-] live_migration data is LibvirtLiveMigrateData(bdms=[],block_migration=True,disk_available_mb=44032,disk_over_commit=<?>,dst_wants_file_backed_memory=False,file_backed_memory_discard=False,filename='tmpdtCugf',graphics_listen_addr_spice=127.0.0.1,graphics_listen_addr_vnc=172.17.1.14,image_type='default',instance_relative_path='92c8e947-31bc-4dd3-aa6e-6666275d9dea',is_shared_block_storage=False,is_shared_instance_path=False,is_volume_backed=False,migration=Migration(c9ac493b-fe1b-44b8-8618-d92a70548d00),old_vol_attachment_ids={},serial_listen_addr=None,serial_listen_ports=[],src_supports_native_luks=True,supported_perf_events=[],target_connect_addr='compute-1.internalapi.localdomain',vifs=[VIFMigrateData],wait_for_vif_plugged=False) _do_live_migration /usr/lib/python2.7/site-packages/nova/compute/manager.py:6436 2019-06-03 23:00:32.199 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Starting monitoring of live migration _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7521 2019-06-03 23:00:32.275 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Operation thread is still running _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7324 2019-06-03 23:00:32.276 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migration not running yet _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7333 2019-06-03 23:00:32.294 1 DEBUG nova.virt.libvirt.migration [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Updating guest XML with vif config: <interface type="bridge"> <mac address="fa:16:3e:db:68:9b"/> <model type="virtio"/> <driver name="vhost" rx_queue_size="512"/> <source bridge="qbr1273b802-b7"/> <mtu size="1450"/> <target dev="tap1273b802-b7"/> </interface> _update_vif_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:318 2019-06-03 23:00:32.295 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] About to invoke the migrate API _live_migration_operation /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7155 2019-06-03 23:00:32.985 1 INFO nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) 2019-06-03 23:00:33.124 1 DEBUG nova.compute.manager [req-f3873904-fc49-476a-883d-89491a57b0b9 6d30bccbef8b4f098c16b1184e710e40 95a7af67e36e4605977726510f8c061f - default default] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Received event network-vif-plugged-1273b802-b7a6-4af0-bb65-5a0e8f7f5902 external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:8217 2019-06-03 23:00:34.129 1 INFO nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migration operation has completed 2019-06-03 23:00:34.129 1 INFO nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] _post_live_migration() is started.. 2019-06-03 23:00:34.130 1 DEBUG os_brick.utils [-] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '172.17.1.19', 'enforce_multipath': True, 'host': 'compute-0.localdomain', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146 2019-06-03 23:00:34.142 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migrate API has completed _live_migration_operation /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7162 2019-06-03 23:00:34.142 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migration operation thread has finished _live_migration_operation /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7211 2019-06-03 23:00:34.142 1 DEBUG nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migration operation thread notification thread_finished /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7512 2019-06-03 23:00:35.910 1 DEBUG nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Calling driver.unfilter_instance from _post_live_migration _post_live_migration /usr/lib/python2.7/site-packages/nova/compute/manager.py:6701 2019-06-03 23:00:37.589 1 DEBUG nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Calling driver.post_live_migration_at_source with original source VIFs from migrate_data: [{"profile": {}, "ovs_interfaceid": "1273b802-b7a6-4af0-bb65-5a0e8f7f5902", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.100.0.4"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "10.100.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.100.0.1"}}], "meta": {"injected": false, "tunneled": true, "tenant_id": "52b68a446d904fa5a2bcc912b29eaafd", "physical_network": null, "mtu": 1450}, "id": "29977de3-fbd7-48d1-b9d9-87beddc2dc6b", "label": "tempest-LiveAutoBlockMigrationV225Test-310507826-network"}, "devname": "tap1273b802-b7", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true}, "address": "fa:16:3e:db:68:9b", "active": false, "type": "ovs", "id": "1273b802-b7a6-4af0-bb65-5a0e8f7f5902", "qbg_params": null}] _post_live_migration /usr/lib/python2.7/site-packages/nova/compute/manager.py:6728 2019-06-03 23:00:38.881 1 DEBUG nova.compute.manager [req-7366d2a1-ea85-47b1-aa9c-3d1b9de259ab 6d30bccbef8b4f098c16b1184e710e40 95a7af67e36e4605977726510f8c061f - default default] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Received event network-vif-unplugged-1273b802-b7a6-4af0-bb65-5a0e8f7f5902 for instance with task_state migrating. 2019-06-03 23:00:41.474 1 DEBUG nova.compute.manager [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Calling driver.cleanup from _post_live_migration _post_live_migration /usr/lib/python2.7/site-packages/nova/compute/manager.py:6761 2019-06-03 23:00:41.475 1 INFO nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Deleting instance files /var/lib/nova/instances/92c8e947-31bc-4dd3-aa6e-6666275d9dea_del 2019-06-03 23:00:41.476 1 INFO nova.virt.libvirt.driver [-] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Deletion of /var/lib/nova/instances/92c8e947-31bc-4dd3-aa6e-6666275d9dea_del complete 2019-06-03 23:00:43.383 1 INFO nova.compute.manager [req-3dd6e091-58d7-46c9-8e6c-73d5a7c52a9b - - - - -] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Migrating instance to compute-1.localdomain finished successfully. 2019-06-03 23:00:43.624 1 INFO nova.scheduler.client.report [req-3dd6e091-58d7-46c9-8e6c-73d5a7c52a9b - - - - -] Deleted allocation for instance c9ac493b-fe1b-44b8-8618-d92a70548d00 2019-06-03 23:00:43.625 1 INFO nova.compute.manager [req-3dd6e091-58d7-46c9-8e6c-73d5a7c52a9b - - - - -] Source node compute-0.localdomain confirmed migration c9ac493b-fe1b-44b8-8618-d92a70548d00; deleted migration-based allocation 2019-06-03 23:00:43.625 1 DEBUG nova.virt.libvirt.driver [req-3dd6e091-58d7-46c9-8e6c-73d5a7c52a9b - - - - -] [instance: 92c8e947-31bc-4dd3-aa6e-6666275d9dea] Live migration monitoring is all done _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7532
We discussed this on the triage call again today. The fact that live migration with: "block_migration=False,is_shared_block_storage=False,is_shared_instance_path=False" fails and live migration with: "block_migration=True,is_shared_block_storage=False,is_shared_instance_path=False" passes implies that NFS is not actually available in the environment. Closing as a duplicate thusly. *** This bug has been marked as a duplicate of bug 1704640 ***