Bug 1385486

Summary: [8.0.z] After upgrading from RHOSP 6 to RHOSP 8 existing instances fail to start.
Product: Red Hat OpenStack Reporter: Lee Yarwood <lyarwood>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: awaugama
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.0 (Liberty)CC: berrange, dasmith, ealcaniz, eglynn, ipetrova, kchamart, lyarwood, mschuppe, pablo.iranzo, panbalag, sbauza, sferdjao, sgordon, srevivo, ssigwald, vaggarwa, vromanso
Target Milestone: asyncKeywords: ZStream
Target Release: 8.0 (Liberty)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-12.0.5-2.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1385404 Environment:
Last Closed: 2016-12-21 16:42:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1385404, 1385500    
Bug Blocks:    
Attachments:
Description Flags
bugfix complementing comment#1 none

Description Lee Yarwood 2016-10-17 07:46:25 UTC 
+++ This bug was initially created as a clone of Bug #1385404 +++

After upgrading to Liberty we tried to restart instances. there are no errors returned. the instance is not starting and the following error shows in the nova-compute.log:

2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher [req-76f17cee-7601-418f-a464-5f1ccf50d9ce admin 048f23a34f524212be242df3b8032a69 - - -] Exception during message handling: Remote error: InvalidTargetVersion Invalid target version 1.2
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 477, in object_class_action_versions\n    if isinstance(result, nova_object.NovaObject) else result)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 514, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.2\n'].
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     payload)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2532, in start_instance
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._power_on(context, instance)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2505, in _power_on
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     block_device_info)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2428, in power_on
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._hard_reboot(context, instance, network_info, block_device_info)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2286, in _hard_reboot
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     write_to_disk=True)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4505, in _get_guest_xml
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     context)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4311, in _get_guest_config
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     instance.numa_topology, flavor, pci_devs, allowed_cpus, image_meta)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 866, in obj_load_attr
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._load_numa_topology()
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 738, in _load_numa_topology
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._context, self.uuid)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 169, in wrapper
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args, kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 229, in object_class_action
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args, kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 237, in object_class_action_versions
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args=args, kwargs=kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     retry=self.retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     retry=retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     raise result
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.2
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 477, in object_class_action_versions\n    if isinstance(result, nova_object.NovaObject) else result)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 514, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.2\n'].
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher

--- Additional comment from Edu Alcaniz on 2016-10-17 03:45:27 EDT ---

We are looking backport to RHOSP8.
Comment 1 Edu Alcaniz 2016-10-17 11:06:06 UTC 
From customer:

Hi again,

We reviewed the patch that Irina provides us (https://git.openstack.org/cgit/openstack/nova/commit/?id=1e3e7309997f90fbd0291c05cc859dd9ac0ef161), and we tried into the controllers of OSP and right now the instances are capable to start. So it works! 

We right now (with the bugfix into the manager.py) are capable or start the instances that had the issue.
Comment 2 Irina Petrova 2016-10-17 11:21:12 UTC 
Created attachment 1211309 [details]
bugfix complementing comment#1

$ tar tvf manager.py.tar.gz 
-rw-rw-r-- ipetrova/ipetrova 48165 2016-10-17 13:00 manager.py
-rw-rw-r-- ipetrova/ipetrova 47364 2016-10-17 13:00 manager.py.bkp
-rw-rw-r-- ipetrova/ipetrova  1885 2016-10-17 13:12 diff.note
Comment 13 errata-xmlrpc 2016-12-21 16:42:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2991.html