Bug 1385404

Summary: [9.0.z] After upgrading from RHOSP 6 to RHOSP 8 existing instances fail to start.
Product: Red Hat OpenStack Reporter: Siggy Sigwald <ssigwald>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: awaugama
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.0 (Liberty)CC: berrange, dasmith, ealcaniz, eglynn, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, vaggarwa, vromanso
Target Milestone: asyncKeywords: ZStream
Target Release: 9.0 (Mitaka)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-13.1.2-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1385486 1385500 (view as bug list) Environment:
Last Closed: 2016-12-21 16:37:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1385500    
Bug Blocks: 1385486    

Description Siggy Sigwald 2016-10-16 21:34:59 UTC 
After upgrading to Liberty we tried to restart instances. there are no errors returned. the instance is not starting and the following error shows in the nova-compute.log:

2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher [req-76f17cee-7601-418f-a464-5f1ccf50d9ce admin 048f23a34f524212be242df3b8032a69 - - -] Exception during message handling: Remote error: InvalidTargetVersion Invalid target version 1.2
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 477, in object_class_action_versions\n    if isinstance(result, nova_object.NovaObject) else result)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 514, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.2\n'].
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     payload)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2532, in start_instance
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._power_on(context, instance)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2505, in _power_on
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     block_device_info)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2428, in power_on
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._hard_reboot(context, instance, network_info, block_device_info)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2286, in _hard_reboot
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     write_to_disk=True)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4505, in _get_guest_xml
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     context)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4311, in _get_guest_config
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     instance.numa_topology, flavor, pci_devs, allowed_cpus, image_meta)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 866, in obj_load_attr
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._load_numa_topology()
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 738, in _load_numa_topology
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._context, self.uuid)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 169, in wrapper
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args, kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 229, in object_class_action
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args, kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 237, in object_class_action_versions
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args=args, kwargs=kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     retry=self.retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     retry=retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     raise result
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.2
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 477, in object_class_action_versions\n    if isinstance(result, nova_object.NovaObject) else result)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 514, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.2\n'].
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher
Comment 1 Edu Alcaniz 2016-10-17 07:45:27 UTC 
We are looking backport to RHOSP8.
Comment 9 errata-xmlrpc 2016-12-21 16:37:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2992.html