Bug 1636280
| Summary: | Instance live-migration sometimes stuck in migrating | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | PURANDHAR SAIRAM MANNIDI <pmannidi> |
| Component: | openstack-nova | Assignee: | Artom Lifshitz <alifshit> |
| Status: | CLOSED ERRATA | QA Contact: | OSP DFG:Compute <osp-dfg-compute> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | akaris, alifshit, berrange, dasmith, dhill, eglynn, jhakimra, kchamart, lyarwood, mbooth, msufiyan, nova-maint, ojanas, pgrist, rchincho, sbauza, sgordon, ssigwald, stephenfin, tvvcox, vromanso |
| Target Milestone: | z11 | Keywords: | Triaged, ZStream |
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-nova-14.1.0-36.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-04-30 16:59:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
PURANDHAR SAIRAM MANNIDI
2018-10-05 00:03:18 UTC
Hi Team, Please update. Sincere Rgds, Sufiyan Hello Artom, I have requested the information from customer, I will update the bugzilla once I get those logs. Thanks, rahulc Ah, I completely missed the compressed log files (.log.gz), and was only looking at plain text files (.log). That being said, as you've already noticed, the conductor logs have no information about request req-a327b195-20cb-4e73-8c46-244660efcaac. At this point I see two options: 1. We could try and obtain full logs, including conductor, that contain a live migration failing in the way described in this bug. 2. I can backport [1] and [2] to OSP10. As I said in comment #7, [2] wouldn't really address the root cause, but it does allow live migrations to complete correctly if there's a race when undefining the domain. I *think* [1] should address the error you're seeing on the destination (unable to set instance task state), but since I've neither reproduced the bug, nor looked at full logs, it's more of a hunch than anything else. If the customer would be willing to accept testing this for us, it could determine whether my hunch is correct. [1] https://review.openstack.org/#/c/389687/ [2] https://review.openstack.org/#/c/508640/ CI has passed for all practical intents and purposes, hotfix can be delivered. *** Bug 1666498 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0923 Making hidden data public after making it anonymous:
=================================
Observations so far:
Instance failed with live migration:
[root@undercloud user]# nova show aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | availability-zone |
| OS-EXT-SRV-ATTR:host | compute-10.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-10.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2018-08-27T21:25:40.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2018-08-27T21:25:29Z |
| fault | {"message": "Conflict updating instance aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa. Expected: {'task_state': [u'migrating']}. Actual: {'task_state': None}", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 204, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 5615, in post_live_migration_at_destination |
| | instance.save(expected_task_state=task_states.MIGRATING) |
| | File \"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py\", line 210, in wrapper |
| | ctxt, self, fn.__name__, args, kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py\", line 241, in object_action |
| | objmethod=objmethod, args=args, kwargs=kwargs) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 169, in call |
| | retry=self.retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 97, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 584, in send |
| | retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 575, in _send |
| | raise result |
| | ", "created": "2018-10-04T12:01:22Z"} |
| flavor | flavor (22) |
| hostId | d5910a0ff31795c10c181cb5591e5a20c5cb0f511e1c440a0483fc44 |
| id | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa |
| image | RHEL (f55eb102-1319-461f-9fdc-702134ab2dad) |
| key_name | ansible |
| metadata | {} |
| name | instance-name |
| os-extended-volumes:volumes_attached | [] |
| security_groups | ICMP, SSH, default |
| status | ERROR |
| tenant-internal-direct-net network | 192.168.123.10 |
| tenant_id | 15b4642e604149eda9dd752285b468cd |
| updated | 2018-10-04T12:02:03Z |
| user_id | 04e0d6bef372166bb29f162d4de79895081a0d9fc919119e6bca9c26f92582b6 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Migration has failed with below traceback in nova-compute on source compute node:
~~~
2018-10-04 12:01:20.454 172213 ERROR nova.virt.libvirt.driver [req-a327b195-20cb-4e73-8c46-244660efcaac 2958242ceb6d46cd8bd84efe9caddba3 f8cdafc8861b4e0ab29362101c9c63ce - - -] [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] Error from libvirt during undefine. Code=42 Error=Domain not found: no domain with matching uuid 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa' (instance-00000001)
2018-10-04 12:01:20.623 172213 WARNING nova.virt.libvirt.driver [req-a327b195-20cb-4e73-8c46-244660efcaac 2958242ceb6d46cd8bd84efe9caddba3 f8cdafc8861b4e0ab29362101c9c63ce - - -] [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] Error monitoring migration: Domain not found: no domain with matching uuid 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa' (instance-00000001)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] Traceback (most recent call last):
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6496, in _live_migration
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] finish_event, disk_paths)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6406, in _live_migration_monitor
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] migrate_data)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] function_name, call_dict, binary)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] self.force_reraise()
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] six.reraise(self.type_, self.value, self.tb)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] return f(self, context, *args, **kw)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] kwargs['instance'], e, sys.exc_info())
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] self.force_reraise()
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] six.reraise(self.type_, self.value, self.tb)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] return function(self, context, *args, **kwargs)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5515, in _post_live_migration
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] destroy_vifs=destroy_vifs)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 978, in cleanup
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] self._undefine_domain(instance)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 876, in _undefine_domain
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] {'errcode': errcode, 'e': e}, instance=instance)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] self.force_reraise()
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] six.reraise(self.type_, self.value, self.tb)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 870, in _undefine_domain
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] guest.delete_configuration(support_uefi)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 277, in delete_configuration
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] self._domain.undefine()
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] rv = execute(f, *args, **kwargs)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] six.reraise(c, e, tb)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] rv = meth(*args, **kwargs)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2615, in undefine
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] if ret == -1: raise libvirtError ('virDomainUndefine() failed', dom=self)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa] libvirtError: Domain not found: no domain with matching uuid 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa' (instance-00000001)
2018-10-04 12:01:20.623 172213 ERROR nova.virt.libvirt.driver [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa]
~~~
What we observed, the instance is running under `virsh list` on destination compute node but under `nova show` instance observed in errored state, and shows source hypervisor under `OS-EXT-SRV-ATTR:hypervisor_hostname`.
Observed following traceback on destination host:
~~~
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server [req-a327b195-20cb-4e73-8c46-244660efcaac 2958242ceb6d46cd8bd84efe9caddba3 f8cdafc8861b4e0
ab29362101c9c63ce - - -] Exception during message handling
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _proc
ess_incoming
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in d
ispatch
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _
do_dispatch
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_rerai
se
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_fu
nction
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_
function
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_rerai
se
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_
function
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5615, in post_live
_migration_at_destination
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server instance.save(expected_task_state=task_states.MIGRATING)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server ctxt, self, fn.__name__, args, kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 241, in object_action
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server objmethod=objmethod, args=args, kwargs=kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server retry=self.retry)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server retry=retry)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server raise result
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server UnexpectedTaskStateError_Remote: Conflict updating instance aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa. Expected: {'task_state': [u'migrating']}. Actual: {'task_state': None}
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 87, in _object_dispatch
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return getattr(target, method)(*args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 761, in save
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server columns_to_join=_expected_cols(expected_attrs))
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 838, in instance_update_and_get_original
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server expected=expected)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2637, in instance_update_and_get_original
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server context, instance_uuid, values, expected, original=instance_ref))
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2773, in _instance_update
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server raise exc(**exc_props)
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server UnexpectedTaskStateError: Conflict updating instance aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa. Expected: {'task_state': [u'migrating']}. Actual: {'task_state': None}
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
2018-10-04 12:01:22.196 26441 ERROR oslo_messaging.rpc.server
~~~
We asked customer to do followings which may resolve the issue:
~~~
nova reset-state --active aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa
nova stop aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa
nova start aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa
~~~
and then try migrating again, customer has tried this workaround before with other errored instances and it may work this time too but they dont want to stop the instance.
Instance is currently reachable and accessible.
|