Bug 1470732 - nova assumes block migration is needed when using ceph volumes for instances and live migration fails
nova assumes block migration is needed when using ceph volumes for instances ...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-novaclient (Show other bugs)
9.0 (Mitaka)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Eoghan Glynn
Joe H. Rahme
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-13 10:24 EDT by Eduard Barrera
Modified: 2017-07-17 08:46 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-17 08:46:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1518059 None None None 2017-07-13 10:24 EDT
Red Hat Knowledge Base (Solution) 3115911 None None None 2017-07-17 08:43 EDT

  None (edit)
Description Eduard Barrera 2017-07-13 10:24:03 EDT
Description of problem:

During the upgrade process from 8 to 9, when live migration of the instances is performed before updating the compute node on a ceph environment, we got the following errors

# nova live-migration [instance-name] [target-hostname] 

2017-07-11 09:58:56.887 22593 WARNING nova.scheduler.utils [req-2ff97d84-084f-4478-b0bf-c19b03c68b94 b21b79c3a5af482cbb3cbc9d5b109493 eae24742f3bb4ed5b442e367a85d321c - - -] Failed to compute_task_migrate_server: overcloud-test-compute-0.localdomain is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
    executor_callback))
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
    executor_callback)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
    payload)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
    kwargs['instance'], e, sys.exc_info())
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5109, in check_can_live_migrate_source
    block_device_info)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5397, in check_can_live_migrate_source
    raise exception.InvalidLocalStorage(reason=reason, path=source)


DEBUG (shell:1082) Live migration with API v2.25 requires all the Mitaka upgrade to be complete before it is available. (HTTP 400) (Request-ID: req-bea5242f-388e-4d70-8d8c-644558d9e583)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1007, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 3853, in do_live_migration
    args.block_migrate)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 442, in live_migrate
    block_migration)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 370, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1530, in live_migrate
    'block_migration': block_migration})
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1682, in _action
    info=info, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1693, in _action_return_resp_and_body
    return self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 179, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request
    raise exceptions.from_response(resp, body, url, method)
BadRequest: Live migration with API v2.25 requires all the Mitaka upgrade to be complete before it is available. (HTTP 400) (Request-ID: req-bea5242f-388e-4d70-8d8c-644558d9e583)
ERROR (BadRequest): Live migration with API v2.25 requires all the Mitaka upgrade to be complete before it is available. (HTTP 400) (Request-ID: req-bea5242f-388e-4d70-8d8c-644558d9e583)


It seems somehow that nova is assuming that ceph volumes should be block migrated an here is when it fails.

Its probably related with [1]


[1] https://bugs.launchpad.net/python-openstackclient/+bug/1518059

Version-Release number of selected component (if applicable):
OSP 9

How reproducible:
always


Steps to Reproduce:
1. on a ceph environment perform live migration of an instance with ceph storage
2.
3.

Actual results:
Live migation fails

Expected results:
Live migration successful 

Additional info:
Comment 3 melanie witt 2017-07-13 23:12:22 EDT
This isn't related to the openstackclient bug [1] because the command 'nova live-migration [instance-name] [target-hostname]' is novaclient, not openstackclient.

It looks like in OSP 9 (Mitaka), a new REST API microversion 2.25 [2] added the ability for an "auto" value for --block-migrate and novaclient with microversion 2.25 defaults --block-migrate to "auto". In the previous microversion 2.24, the --block-migrate option defaulted to False.

The error message is saying that if you make a request with REST API microversion 2.25 and with --block-migrate "auto" it will be rejected until all computes are upgraded to OSP 9 (Mitaka).

To make it work, you either need to issue the command passing False for block live migration:

  'nova live-migration --block-migrate False [instance-name] [target-hostname]'

or specify the earlier REST API microversion 2.24 when you issue the command:

  'nova --os-compute-api-version 2.24 nova live-migration [instance-name] [target-hostname]'


[1] https://bugs.launchpad.net/python-openstackclient/+bug/1518059
[2] https://docs.openstack.org/nova/latest/api_microversion_history.html#maximum-in-mitaka
[3] https://github.com/openstack/python-novaclient/blob/mitaka-eol/novaclient/v2/shell.py#L3808-L3821
Comment 4 melanie witt 2017-07-14 10:36:03 EDT
(In reply to melanie witt from comment #3)
>   'nova --os-compute-api-version 2.24 nova live-migration [instance-name]
> [target-hostname]'

Sorry, there's accidentally an extra 'nova' in the above command. It should be:

  'nova --os-compute-api-version 2.24 live-migration [instance-name] [target-hostname]'
Comment 5 melanie witt 2017-07-14 10:41:34 EDT
(In reply to melanie witt from comment #3)
>   'nova live-migration --block-migrate False [instance-name]
> [target-hostname]'

And apparently the above won't work because the --block-migrate option is a 'store_true' argument and there's not a way to switch it to False.

So to amend my original comment, to make live migration work in this situation, you must specify the earlier REST API microversion 2.24 when you issue the command:

  'nova --os-compute-api-version 2.24 live-migration [instance-name] [target-hostname]'
Comment 6 Martin Schuppert 2017-07-17 08:46:16 EDT
We have documented this in https://access.redhat.com/solutions/3115911 . Closing this BZ.

Note You need to log in before you can comment on or make changes to this bug.