1385486 – [8.0.z] After upgrading from RHOSP 6 to RHOSP 8 existing instances fail to start.

Bug 1385486 - [8.0.z] After upgrading from RHOSP 6 to RHOSP 8 existing instances fail to start.

Summary: [8.0.z] After upgrading from RHOSP 6 to RHOSP 8 existing instances fail to st...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	8.0 (Liberty)
Hardware:	All
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	async
Target Release:	8.0 (Liberty)
Assignee:	Lee Yarwood
QA Contact:	awaugama
Docs Contact:
URL:
Whiteboard:
Depends On:	1385404 1385500
Blocks:
TreeView+	depends on / blocked

Reported:	2016-10-17 07:46 UTC by Lee Yarwood
Modified:	2022-07-09 08:34 UTC (History)
CC List:	17 users (show)
Fixed In Version:	openstack-nova-12.0.5-2.el7ost
Doc Type:	Bug Fix
Doc Text:
Clone Of:	1385404
Environment:
Last Closed:	2016-12-21 16:42:06 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
bugfix complementing comment#1 (17.93 KB, application/x-gzip) 2016-10-17 11:21 UTC, Irina Petrova	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1596119	None	None	None	2016-10-17 07:46:24 UTC
Red Hat Issue Tracker	OSP-16787	None	None	None	2022-07-09 08:34:12 UTC
Red Hat Knowledge Base (Solution)	2708841	None	None	None	2016-10-17 07:46:24 UTC
Red Hat Product Errata	RHSA-2016:2991	normal	SHIPPED_LIVE	Moderate: openstack-cinder, openstack-glance, and openstack-nova update	2016-12-21 21:34:31 UTC

Description Lee Yarwood 2016-10-17 07:46:25 UTC

+++ This bug was initially created as a clone of Bug #1385404 +++

After upgrading to Liberty we tried to restart instances. there are no errors returned. the instance is not starting and the following error shows in the nova-compute.log:

2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher [req-76f17cee-7601-418f-a464-5f1ccf50d9ce admin 048f23a34f524212be242df3b8032a69 - - -] Exception during message handling: Remote error: InvalidTargetVersion Invalid target version 1.2
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 477, in object_class_action_versions\n    if isinstance(result, nova_object.NovaObject) else result)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 514, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.2\n'].
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     payload)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2532, in start_instance
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._power_on(context, instance)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2505, in _power_on
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     block_device_info)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2428, in power_on
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._hard_reboot(context, instance, network_info, block_device_info)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2286, in _hard_reboot
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     write_to_disk=True)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4505, in _get_guest_xml
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     context)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4311, in _get_guest_config
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     instance.numa_topology, flavor, pci_devs, allowed_cpus, image_meta)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 866, in obj_load_attr
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._load_numa_topology()
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 738, in _load_numa_topology
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     self._context, self.uuid)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 169, in wrapper
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args, kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 229, in object_class_action
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args, kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 237, in object_class_action_versions
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     args=args, kwargs=kwargs)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     retry=self.retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     retry=retry)
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher     raise result
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.2
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 477, in object_class_action_versions\n    if isinstance(result, nova_object.NovaObject) else result)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 514, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.2\n'].
2016-10-16 11:45:23.316 26103 ERROR oslo_messaging.rpc.dispatcher

--- Additional comment from Edu Alcaniz on 2016-10-17 03:45:27 EDT ---

We are looking backport to RHOSP8.

Comment 1 Edu Alcaniz 2016-10-17 11:06:06 UTC

From customer:

Hi again,

We reviewed the patch that Irina provides us (https://git.openstack.org/cgit/openstack/nova/commit/?id=1e3e7309997f90fbd0291c05cc859dd9ac0ef161), and we tried into the controllers of OSP and right now the instances are capable to start. So it works! 

We right now (with the bugfix into the manager.py) are capable or start the instances that had the issue.

Comment 2 Irina Petrova 2016-10-17 11:21:12 UTC

Created attachment 1211309 [details]
bugfix complementing comment#1

$ tar tvf manager.py.tar.gz 
-rw-rw-r-- ipetrova/ipetrova 48165 2016-10-17 13:00 manager.py
-rw-rw-r-- ipetrova/ipetrova 47364 2016-10-17 13:00 manager.py.bkp
-rw-rw-r-- ipetrova/ipetrova  1885 2016-10-17 13:12 diff.note

Comment 13 errata-xmlrpc 2016-12-21 16:42:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2991.html

Note You need to log in before you can comment on or make changes to this bug.