Bug 1470732
| Summary: | nova assumes block migration is needed when using ceph volumes for instances and live migration fails | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eduard Barrera <ebarrera> |
| Component: | python-novaclient | Assignee: | Eoghan Glynn <eglynn> |
| Status: | CLOSED NOTABUG | QA Contact: | Joe H. Rahme <jhakimra> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 9.0 (Mitaka) | CC: | awaugama, berrange, dasmith, eglynn, jruzicka, kchamart, mschuppe, mwitt, pablo.iranzo, sbauza, sferdjao, sgordon, srevivo, vromanso |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-07-17 12:46:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This isn't related to the openstackclient bug [1] because the command 'nova live-migration [instance-name] [target-hostname]' is novaclient, not openstackclient. It looks like in OSP 9 (Mitaka), a new REST API microversion 2.25 [2] added the ability for an "auto" value for --block-migrate and novaclient with microversion 2.25 defaults --block-migrate to "auto". In the previous microversion 2.24, the --block-migrate option defaulted to False. The error message is saying that if you make a request with REST API microversion 2.25 and with --block-migrate "auto" it will be rejected until all computes are upgraded to OSP 9 (Mitaka). To make it work, you either need to issue the command passing False for block live migration: 'nova live-migration --block-migrate False [instance-name] [target-hostname]' or specify the earlier REST API microversion 2.24 when you issue the command: 'nova --os-compute-api-version 2.24 nova live-migration [instance-name] [target-hostname]' [1] https://bugs.launchpad.net/python-openstackclient/+bug/1518059 [2] https://docs.openstack.org/nova/latest/api_microversion_history.html#maximum-in-mitaka [3] https://github.com/openstack/python-novaclient/blob/mitaka-eol/novaclient/v2/shell.py#L3808-L3821 (In reply to melanie witt from comment #3) > 'nova --os-compute-api-version 2.24 nova live-migration [instance-name] > [target-hostname]' Sorry, there's accidentally an extra 'nova' in the above command. It should be: 'nova --os-compute-api-version 2.24 live-migration [instance-name] [target-hostname]' (In reply to melanie witt from comment #3) > 'nova live-migration --block-migrate False [instance-name] > [target-hostname]' And apparently the above won't work because the --block-migrate option is a 'store_true' argument and there's not a way to switch it to False. So to amend my original comment, to make live migration work in this situation, you must specify the earlier REST API microversion 2.24 when you issue the command: 'nova --os-compute-api-version 2.24 live-migration [instance-name] [target-hostname]' We have documented this in https://access.redhat.com/solutions/3115911 . Closing this BZ. |
Description of problem: During the upgrade process from 8 to 9, when live migration of the instances is performed before updating the compute node on a ceph environment, we got the following errors # nova live-migration [instance-name] [target-hostname] 2017-07-11 09:58:56.887 22593 WARNING nova.scheduler.utils [req-2ff97d84-084f-4478-b0bf-c19b03c68b94 b21b79c3a5af482cbb3cbc9d5b109493 eae24742f3bb4ed5b442e367a85d321c - - -] Failed to compute_task_migrate_server: overcloud-test-compute-0.localdomain is not on local storage: Block migration can not be used with shared storage. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply executor_callback)) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch executor_callback) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch result = func(ctxt, **new_args) File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped payload) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped return f(self, context, *args, **kw) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function kwargs['instance'], e, sys.exc_info()) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5109, in check_can_live_migrate_source block_device_info) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5397, in check_can_live_migrate_source raise exception.InvalidLocalStorage(reason=reason, path=source) DEBUG (shell:1082) Live migration with API v2.25 requires all the Mitaka upgrade to be complete before it is available. (HTTP 400) (Request-ID: req-bea5242f-388e-4d70-8d8c-644558d9e583) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1007, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 3853, in do_live_migration args.block_migrate) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 442, in live_migrate block_migration) File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 370, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1530, in live_migrate 'block_migration': block_migration}) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1682, in _action info=info, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1693, in _action_return_resp_and_body return self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 179, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request raise exceptions.from_response(resp, body, url, method) BadRequest: Live migration with API v2.25 requires all the Mitaka upgrade to be complete before it is available. (HTTP 400) (Request-ID: req-bea5242f-388e-4d70-8d8c-644558d9e583) ERROR (BadRequest): Live migration with API v2.25 requires all the Mitaka upgrade to be complete before it is available. (HTTP 400) (Request-ID: req-bea5242f-388e-4d70-8d8c-644558d9e583) It seems somehow that nova is assuming that ceph volumes should be block migrated an here is when it fails. Its probably related with [1] [1] https://bugs.launchpad.net/python-openstackclient/+bug/1518059 Version-Release number of selected component (if applicable): OSP 9 How reproducible: always Steps to Reproduce: 1. on a ceph environment perform live migration of an instance with ceph storage 2. 3. Actual results: Live migation fails Expected results: Live migration successful Additional info: