Description of problem: openstack server migrate --live fails with the following error: [stack@undercloud-0 ~]$ openstack server migrate st--d0-instance-grw4dxoglahw-my_instance-4cm3la6a6xba --live overcloud-compute-0.localdomain overcloud-compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-6fe46746-c5fc-414a-8bbb-f7e056965396) The nova instances are stored on Ceph storage so I expect the live migration to work. nova live-migration st--d0-instance-grw4dxoglahw-my_instance-4cm3la6a6xba overcloud-compute-0.localdomain works fine instead. Version-Release number of selected component (if applicable): python-openstackclient-3.2.1-1.el7ost.noarch python-novaclient-6.0.0-1.el7ost.noarch How reproducible: 100%
*** Bug 1424066 has been marked as a duplicate of this bug. ***
(In reply to Marius Cornea from comment #0) > Description of problem: > openstack server migrate --live fails with the following error: > > [stack@undercloud-0 ~]$ openstack server migrate > st--d0-instance-grw4dxoglahw-my_instance-4cm3la6a6xba --live > overcloud-compute-0.localdomain > overcloud-compute-1.localdomain is not on local storage: Block migration can > not be used with shared storage. (HTTP 400) (Request-ID: > req-6fe46746-c5fc-414a-8bbb-f7e056965396) > > The nova instances are stored on Ceph storage so I expect the live migration > to work. > > nova live-migration st--d0-instance-grw4dxoglahw-my_instance-4cm3la6a6xba > overcloud-compute-0.localdomain works fine instead. > > Version-Release number of selected component (if applicable): > python-openstackclient-3.2.1-1.el7ost.noarch > python-novaclient-6.0.0-1.el7ost.noarch > > How reproducible: > 100% That is little bit confusing, the help message for the command "openstack server migrate" is saying that the default behavior is to use "shared-migration" [0] but the error message returned makes me think a "block-migration" has been initiated. Without any log is difficult to say. So first can you try to execute this same command but specify that you want to use "shared-migration", something like: $ openstack server migrate st--d0-instance-grw4dxoglahw-my_instance-4cm3la6a6xba --shared-migration --live overcloud-compute-0.localdomain [0] https://docs.openstack.org/developer/python-openstackclient/command-objects/server.html#server-migrate
Hi Sahid, Please can you or anyone check whether the shared-migration vs block-migration are not switched over? When migrating the instance with local ephemeral disks one seems to must use "--shared-migration" When migrating the instance that is boot from a volume that is shared one seems to must use "--block-migration" Thanks!
(In reply to Sahid Ferdjaoui from comment #2) [..] > $ openstack server migrate > st--d0-instance-grw4dxoglahw-my_instance-4cm3la6a6xba --shared-migration > --live overcloud-compute-0.localdomain It has the same result: [stack@undercloud-0 ~]$ openstack server migrate st--b2-instance-2czjr4x7peym-my_instance-rhppwfsbddpd --shared-migration --live overcloud-compute-1.localdomain overcloud-compute-0.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-40cc689c-66d4-4640-84da-128e0d9b869b) > [0] > https://docs.openstack.org/developer/python-openstackclient/command-objects/ > server.html#server-migrate
Created attachment 1272239 [details] nova-conductor.log Attaching the nova-conductor.log
I'm seeing the same. openstack server migrate --shared-migration --live compute200.cluster 6da6035b-fc96-414e-a0d0-99e057bd90e7 compute232.cluster is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-7c9c111b-e9b0-4a05-beff-2638bca81f3f)
This seem to be an issue with the openstackclient: [stack@undercloud-0 ~]$ openstack server show test +--------------------------------------+----------------------------------------------------------+ | Field | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-1.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-07-05T12:31:27.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | private=192.168.0.9 | | config_drive | | | created | 2017-07-05T12:31:14Z | | flavor | m1.small (d8a898c0-5ed1-484f-843b-96558efd7dbf) | | hostId | 1013a9907516d64716b1db5ccec36a67262b249f24f1173f0bd9eeef | | id | 11f76706-c034-47c7-a40e-bb223ec7dd2f | | image | cirros (27ede6d9-96e2-4dd1-9173-8ef566e60a26) | | key_name | None | | name | test | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | project_id | 6eeb1132ac7b43d7a97fc2b4eb82d461 | | properties | | | security_groups | [{u'name': u'default'}] | | status | ACTIVE | | updated | 2017-07-05T12:34:11Z | | user_id | 7fab49be19df4b499eeaf6bbd67bc93c | +--------------------------------------+----------------------------------------------------------+ [stack@undercloud-0 ~]$ openstack server migrate test --live compute-0.localdomain compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-e1713e69-e05f-4e4f-8582-f7dc64210a62) From nova API log: 2017-07-05 12:36:40.323 5178 DEBUG nova.api.openstack.wsgi [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Action: 'action', calling method: <bound method MigrateServerController._migrate_live of <nova.api.openstack.compute.migrate_server.MigrateServerController object at 0x8d272d0>>, body: {"os-migrateLive": {"disk_over_commit": false, "block_migration": true, "host": "compute-0.localdomain"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:633 2017-07-05 12:36:40.325 5178 DEBUG nova.compute.api [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] [instance: 11f76706-c034-47c7-a40e-bb223ec7dd2f] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2213 2017-07-05 12:36:40.400 5178 DEBUG nova.compute.api [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] [instance: 11f76706-c034-47c7-a40e-bb223ec7dd2f] Going to try to live migrate instance to compute-0.localdomain live_migrate /usr/lib/python2.7/site-packages/nova/compute/api.py:3621 2017-07-05 12:36:40.480 5178 DEBUG nova.objects.instance [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Lazy-loading 'flavor' on Instance uuid 11f76706-c034-47c7-a40e-bb223ec7dd2f obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2017-07-05 12:36:40.557 5178 WARNING oslo_config.cfg [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal. Its value may be silently ignored in the future. 2017-07-05 12:36:40.557 5178 WARNING oslo_config.cfg [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal. Its value may be silently ignored in the future. 2017-07-05 12:36:40.558 5178 WARNING oslo_config.cfg [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal. Its value may be silently ignored in the future. 2017-07-05 12:36:40.577 5178 DEBUG oslo_messaging._drivers.amqpdriver [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 626c1a469b164b7e857cf01e82132fbd NOTIFY exchange 'nova' topic 'notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:479 2017-07-05 12:36:40.589 5178 DEBUG oslo_messaging._drivers.amqpdriver [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 360181b07a604cf5a510aae14ade40ec NOTIFY exchange 'nova' topic 'versioned_notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:479 2017-07-05 12:36:40.685 5178 DEBUG oslo_messaging._drivers.amqpdriver [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CALL msg_id: 1106eb63241141468719d706b2b13852 exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:495 2017-07-05 12:36:41.486 5178 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 1106eb63241141468719d706b2b13852 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:343 2017-07-05 12:36:41.487 5178 INFO nova.api.openstack.wsgi [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] HTTP exception thrown: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. 2017-07-05 12:36:41.488 5178 DEBUG nova.api.openstack.wsgi [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Returning 400 to user: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044 2017-07-05 12:36:41.489 5178 INFO nova.osapi_compute.wsgi.server [req-e1713e69-e05f-4e4f-8582-f7dc64210a62 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] 172.17.1.19 "POST /v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f/action HTTP/1.1" status: 400 len: 476 time: 1.1738911 -> we see '"block_migration": true' even the default is false Also when we explicitly specify shared-migration we have the '"block_migration": true' [stack@undercloud-0 ~]$ openstack server migrate test --shared-migration --live compute-0.localdomain --debug START with options: [u'server', u'migrate', u'test', u'--shared-migration', u'--live', u'compute-0.localdomain', u'--debug'] options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='', auth_url='http://10.0.0.106:5000/v2.0', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, murano_url='', old_profile=None, openid_scope='', os_alarming_api_version='2', os_application_catalog_api_version='1', os_baremetal_api_version='1.6', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_dns_api_version='2', os_identity_api_version='', os_image_api_version='', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.106:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.6', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.106:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.6', 'queues_api_version': '2', 'auth': {'username': 'admin', 'project_name': 'admin', 'password': '***', 'auth_url': 'http://10.0.0.106:5000/v2.0'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} compute API version 2, cmd group openstack.compute.v2 network API version 2, cmd group openstack.network.v2 image API version 2, cmd group openstack.image.v2 volume API version 2, cmd group openstack.volume.v2 identity API version 2.0, cmd group openstack.identity.v2 object_store API version 1, cmd group openstack.object_store.v1 neutronclient API version 2, cmd group openstack.neutronclient.v2 messaging API version 2, cmd group openstack.messaging.v2 orchestration API version 1, cmd group openstack.orchestration.v1 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 baremetal API version 1.6, cmd group openstack.baremetal.v1 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 application_catalog API version 1, cmd group openstack.application_catalog.v1 key_manager API version 1, cmd group openstack.key_manager.v1 container_infra API version 1, cmd group openstack.container_infra.v1 dns API version 2, cmd group openstack.dns.v2 alarming API version 2, cmd group openstack.alarming.v2 metric API version 1, cmd group openstack.metric.v1 data_processing API version 1.1, cmd group openstack.data_processing.v1 Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.106:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.6', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.106:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.6', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} command: server migrate -> openstackclient.compute.v2.server.MigrateServer Using auth plugin: password Using parameters {'username': 'admin', 'project_name': 'admin', 'password': '***', 'auth_url': 'http://10.0.0.106:5000/v2.0'} Get auth_ref REQ: curl -g -i -X GET http://10.0.0.106:5000/v2.0 -H "Accept: application/json" -H "User-Agent: osc-lib keystoneauth1/2.12.3 python-requests/2.10.0 CPython/2.7.5" Starting new HTTP connection (1): 10.0.0.106 "GET /v2.0 HTTP/1.1" 200 227 RESP: [200] Date: Wed, 05 Jul 2017 12:47:55 GMT Server: Apache Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-bd670615-a76c-4fc2-b271-fec659b8e1b4 Content-Encoding: gzip Content-Length: 227 Content-Type: application/json RESP BODY: {"version": {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.0.0.106:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}} Making authentication request to http://10.0.0.106:5000/v2.0/tokens "POST /v2.0/tokens HTTP/1.1" 200 1087 run(Namespace(disk_overcommit=False, live=u'compute-0.localdomain', server=u'test', shared_migration=True, wait=False)) Instantiating compute client for API Version Major: 2, Minor: 0 Making authentication request to http://10.0.0.106:5000/v2.0/tokens "POST /v2.0/tokens HTTP/1.1" 200 1085 REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1/servers/test -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}be7810a2c50fc9b2baba8f2ad3b16b8359876f63" Starting new HTTP connection (1): 10.0.0.106 "GET /v2.1/servers/test HTTP/1.1" 404 79 RESP: [404] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 79 X-Compute-Request-Id: req-7ef45fe8-6622-4780-9d78-85202c597e33 Date: Wed, 05 Jul 2017 12:47:56 GMT RESP BODY: {"itemNotFound": {"message": "Instance test could not be found.", "code": 404}} GET call to compute for http://10.0.0.106:8774/v2.1/servers/test used request id req-7ef45fe8-6622-4780-9d78-85202c597e33 REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1/servers/test -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}be7810a2c50fc9b2baba8f2ad3b16b8359876f63" "GET /v2.1/servers/test HTTP/1.1" 404 79 RESP: [404] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 79 X-Compute-Request-Id: req-7440e916-3a4c-4f7c-9d88-b6eb9b10e90e Date: Wed, 05 Jul 2017 12:47:56 GMT RESP BODY: {"itemNotFound": {"message": "Instance test could not be found.", "code": 404}} GET call to compute for http://10.0.0.106:8774/v2.1/servers/test used request id req-7440e916-3a4c-4f7c-9d88-b6eb9b10e90e REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1/servers?name=test -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}be7810a2c50fc9b2baba8f2ad3b16b8359876f63" "GET /v2.1/servers?name=test HTTP/1.1" 200 289 RESP: [200] Content-Length: 289 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-f9cd6b2e-a9b0-405c-a4c2-4c90b0d073ca Date: Wed, 05 Jul 2017 12:47:56 GMT RESP BODY: {"servers": [{"id": "11f76706-c034-47c7-a40e-bb223ec7dd2f", "links": [{"href": "http://10.0.0.106:8774/v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f", "rel": "self"}, {"href": "http://10.0.0.106:8774/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f", "rel": "bookmark"}], "name": "test"}]} GET call to compute for http://10.0.0.106:8774/v2.1/servers?name=test used request id req-f9cd6b2e-a9b0-405c-a4c2-4c90b0d073ca REQ: curl -g -i -X GET http://10.0.0.106:8774/v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}be7810a2c50fc9b2baba8f2ad3b16b8359876f63" "GET /v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f HTTP/1.1" 200 1632 RESP: [200] Content-Length: 1632 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-fd2515ae-5107-4934-8be4-7b9af23a6850 Date: Wed, 05 Jul 2017 12:47:57 GMT RESP BODY: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:b4:b4:96", "version": 4, "addr": "192.168.0.9", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.0.0.106:8774/v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f", "rel": "self"}, {"href": "http://10.0.0.106:8774/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f", "rel": "bookmark"}], "image": {"id": "27ede6d9-96e2-4dd1-9173-8ef566e60a26", "links": [{"href": "http://10.0.0.106:8774/images/27ede6d9-96e2-4dd1-9173-8ef566e60a26", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000003", "OS-SRV-USG:launched_at": "2017-07-05T12:31:27.000000", "flavor": {"id": "d8a898c0-5ed1-484f-843b-96558efd7dbf", "links": [{"href": "http://10.0.0.106:8774/flavors/d8a898c0-5ed1-484f-843b-96558efd7dbf", "rel": "bookmark"}]}, "id": "11f76706-c034-47c7-a40e-bb223ec7dd2f", "security_groups": [{"name": "default"}], "user_id": "7fab49be19df4b499eeaf6bbd67bc93c", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2017-07-05T12:45:18Z", "hostId": "1013a9907516d64716b1db5ccec36a67262b249f24f1173f0bd9eeef", "OS-EXT-SRV-ATTR:host": "compute-1.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute-1.localdomain", "name": "test", "created": "2017-07-05T12:31:14Z", "tenant_id": "6eeb1132ac7b43d7a97fc2b4eb82d461", "os-extended-volumes:volumes_attached": [], "config_drive": ""}} GET call to compute for http://10.0.0.106:8774/v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f used request id req-fd2515ae-5107-4934-8be4-7b9af23a6850 REQ: curl -g -i -X POST http://10.0.0.106:8774/v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f/action -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}be7810a2c50fc9b2baba8f2ad3b16b8359876f63" -d '{"os-migrateLive": {"disk_over_commit": false, "block_migration": true, "host": "compute-0.localdomain"}}' "POST /v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f/action HTTP/1.1" 400 143 RESP: [400] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 143 X-Compute-Request-Id: req-1c9382d5-0495-431c-a9b6-56af2a027a0a Date: Wed, 05 Jul 2017 12:47:58 GMT--- RESP BODY: {"badRequest": {"message": "compute-1.localdomain is not on local storage: Block migration can not be used with shared storage.", "code": 400}} POST call to compute for http://10.0.0.106:8774/v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f/action used request id req-1c9382d5-0495-431c-a9b6-56af2a027a0a compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-1c9382d5-0495-431c-a9b6-56af2a027a0a) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run return self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 1021, in take_action parsed_args.disk_overcommit, File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 433, in live_migrate disk_over_commit)--- File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1635, in live_migrate 'disk_over_commit': disk_over_commit}) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1845, in _action info=info, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1856, in _action_return_resp_and_body return self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 193, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 178, in request raise exceptions.from_response(resp, body, url, method) BadRequest: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-1c9382d5-0495-431c-a9b6-56af2a027a0a) clean_up MigrateServer: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-1c9382d5-0495-431c-a9b6-56af2a027a0a) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run return self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 1021, in take_action parsed_args.disk_overcommit, File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 433, in live_migrate disk_over_commit)--- File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1635, in live_migrate 'disk_over_commit': disk_over_commit}) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1845, in _action info=info, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1856, in _action_return_resp_and_body return self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 193, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 178, in request raise exceptions.from_response(resp, body, url, method) BadRequest: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. (HTTP 400) (Request-ID: req-1c9382d5-0495-431c-a9b6-56af2a027a0a) END return value: 1 2017-07-05 12:47:57.255 5177 DEBUG nova.api.openstack.wsgi [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Action: 'action', calling method: <bound method MigrateServerController._migrate_live of <nova.api.openstack.compute.migrate_server.MigrateServerController object at 0x8d272d0>>, body: {"os-migrateLive": {"disk_over_commit": false, "block_migration": true, "host": "compute-0.localdomain"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:633 2017-07-05 12:47:57.256 5177 DEBUG nova.compute.api [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] [instance: 11f76706-c034-47c7-a40e-bb223ec7dd2f] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2213 2017-07-05 12:47:57.336 5177 DEBUG nova.compute.api [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] [instance: 11f76706-c034-47c7-a40e-bb223ec7dd2f] Going to try to live migrate instance to compute-0.localdomain live_migrate /usr/lib/python2.7/site-packages/nova/compute/api.py:3621 2017-07-05 12:47:57.417 5177 DEBUG nova.objects.instance [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Lazy-loading 'flavor' on Instance uuid 11f76706-c034-47c7-a40e-bb223ec7dd2f obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2017-07-05 12:47:57.684 5177 DEBUG oslo_messaging._drivers.amqpdriver [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 4ec96e79baa745cc92359351d2b45f0e NOTIFY exchange 'nova' topic 'notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:479 2017-07-05 12:47:57.691 5177 DEBUG oslo_messaging._drivers.amqpdriver [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 79a939ad16344ce6a46a36b59ceeec85 NOTIFY exchange 'nova' topic 'versioned_notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:479 2017-07-05 12:47:57.727 5177 DEBUG oslo_messaging._drivers.amqpdriver [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CALL msg_id: 4566a1e4c05f4fad805a631f62c25159 exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:495 2017-07-05 12:47:58.486 5177 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 4566a1e4c05f4fad805a631f62c25159 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:343 2017-07-05 12:47:58.489 5177 INFO nova.api.openstack.wsgi [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] HTTP exception thrown: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. 2017-07-05 12:47:58.490 5177 DEBUG nova.api.openstack.wsgi [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Returning 400 to user: compute-1.localdomain is not on local storage: Block migration can not be used with shared storage. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044 2017-07-05 12:47:58.492 5177 INFO nova.osapi_compute.wsgi.server [req-1c9382d5-0495-431c-a9b6-56af2a027a0a 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] 172.17.1.19 "POST /v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f/action HTTP/1.1" status: 400 len: 476 time: 1.2447522 Note - the same works with the novaclient: [stack@undercloud-0 ~]$ nova live-migration test compute-1.localdomain [stack@undercloud-0 ~]$ openstack server show test +--------------------------------------+----------------------------------------------------------+ | Field | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-1.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-07-05T12:31:27.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | private=192.168.0.9 | | config_drive | | | created | 2017-07-05T12:31:14Z | | flavor | m1.small (d8a898c0-5ed1-484f-843b-96558efd7dbf) | | hostId | cb3102621af3618ae1cb7b0c97eeadea9010da9584efe910c4afd58c | | id | 11f76706-c034-47c7-a40e-bb223ec7dd2f | | image | cirros (27ede6d9-96e2-4dd1-9173-8ef566e60a26) | | key_name | None | | name | test | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | project_id | 6eeb1132ac7b43d7a97fc2b4eb82d461 | | properties | | | security_groups | [{u'name': u'default'}] | | status | ACTIVE | | updated | 2017-07-05T13:01:18Z | | user_id | 7fab49be19df4b499eeaf6bbd67bc93c | +--------------------------------------+----------------------------------------------------------+ 2017-07-05 13:29:40.404 5176 DEBUG nova.api.openstack.wsgi [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Action: 'action', calling method: <bound method MigrateServerController._migrate_live of <nova.api.openstack.compute.migrate_server.MigrateServerController object at 0x8d272d0>>, body: {"os-migrateLive": {"block_migration": "auto", "host": "compute-1.localdomain"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:633 2017-07-05 13:29:40.405 5176 DEBUG nova.compute.api [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] [instance: 11f76706-c034-47c7-a40e-bb223ec7dd2f] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2213 2017-07-05 13:29:40.488 5176 DEBUG nova.compute.api [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] [instance: 11f76706-c034-47c7-a40e-bb223ec7dd2f] Going to try to live migrate instance to compute-1.localdomain live_migrate /usr/lib/python2.7/site-packages/nova/compute/api.py:3621 2017-07-05 13:29:40.578 5176 DEBUG nova.objects.instance [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] Lazy-loading 'flavor' on Instance uuid 11f76706-c034-47c7-a40e-bb223ec7dd2f obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2017-07-05 13:29:40.660 5176 DEBUG oslo_messaging._drivers.amqpdriver [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 364bf199782840929fb581bd7f1e2755 NOTIFY exchange 'nova' topic 'notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:479 2017-07-05 13:29:40.668 5176 DEBUG oslo_messaging._drivers.amqpdriver [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 1657e4f62bc94b25aedded24bffe9bb5 NOTIFY exchange 'nova' topic 'versioned_notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:479 2017-07-05 13:29:40.708 5176 DEBUG oslo_messaging._drivers.amqpdriver [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] CAST unique_id: 1bda400af08d46d19ae3888f03560533 exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:495 2017-07-05 13:29:40.713 5176 INFO nova.osapi_compute.wsgi.server [req-8da9379b-29db-445a-b58a-9c01b6bf4396 7fab49be19df4b499eeaf6bbd67bc93c 6eeb1132ac7b43d7a97fc2b4eb82d461 - default default] 172.17.1.19 "POST /v2.1/servers/11f76706-c034-47c7-a40e-bb223ec7dd2f/action HTTP/1.1" status: 202 len: 315 time: 0.3172231 here we see '"block_migration": "auto"'
Fixed upstream with https://review.openstack.org/#/c/385138/ . After applying that patch live migration with shared storage (tested with ceph) works.
Nice debugging Martin. Have cherry-picked change upstream, if it helps.
Backport has landed upstream so perhaps a Red Hatter can pull this into RHOSP please?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1596