Bug 1403380 - openstack-nova: nova host-servers-migrate [hostname] fails when the target node is upgraded (during upgrade osp9->osp10)
Summary: openstack-nova: nova host-servers-migrate [hostname] fails when the target n...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: z2
: 10.0 (Newton)
Assignee: Dan Smith
QA Contact: Prasanth Anbalagan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-09 21:19 UTC by Alexander Chuzhoy
Modified: 2019-09-09 13:26 UTC (History)
17 users (show)

Fixed In Version: openstack-nova-14.0.2-10.el7ost
Doc Type: Bug Fix
Doc Text:
The `Instance.pci_devices` field is nullable, but previously the `get_instance_pci_devs()` function did not account for that possibility. If it was passed to an instance from an older deployment (for example Red Hat OpenStack Platform 9) with no `pci_devices`, it would crash trying to iterate `None` as a list. This issue also affected live migration triggered as part of the upgrade workflow; specifically, attempting to live migrate guests from Red Hat OpenStack Platform 9 hosts to Red Hat OpenStack Platform 10 hosts. The issue is now resolved in Red Hat OpenStack Platform 10 as of this update.
Clone Of:
Environment:
Last Closed: 2017-02-23 21:14:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1648887 0 None None None 2016-12-09 21:23:02 UTC
OpenStack gerrit 409821 0 None None None 2016-12-14 17:51:14 UTC
Red Hat Product Errata RHBA-2017:0319 0 normal SHIPPED_LIVE openstack-nova bug fix advisory 2017-02-24 02:12:47 UTC

Description Alexander Chuzhoy 2016-12-09 21:19:42 UTC 
openstack-nova: nova host-servers-migrate [hostname] fails  when the target node is upgraded (during upgrade osp9->osp10)
Environment:
openstack-tripleo-heat-templates-5.1.0-7.el7ost.noarch
openstack-nova-scheduler-14.0.2-7.el7ost.noarch
openstack-nova-conductor-14.0.2-7.el7ost.noarch
openstack-nova-compute-14.0.2-7.el7ost.noarch
puppet-nova-9.4.0-1.el7ost.noarch
openstack-puppet-modules-9.3.0-1.el7ost.noarch
instack-undercloud-5.1.0-4.el7ost.noarch
openstack-nova-network-14.0.2-7.el7ost.noarch
openstack-nova-cert-14.0.2-7.el7ost.noarch
python-nova-tests-14.0.2-7.el7ost.noarch
python-nova-14.0.2-7.el7ost.noarch
openstack-nova-placement-api-14.0.2-7.el7ost.noarch
openstack-nova-console-14.0.2-7.el7ost.noarch
openstack-nova-novncproxy-14.0.2-7.el7ost.noarch
python-novaclient-6.0.0-1.el7ost.noarch
openstack-nova-api-14.0.2-7.el7ost.noarch
openstack-tripleo-heat-templates-compat-2.0.0-41.el7ost.noarch
openstack-nova-cells-14.0.2-7.el7ost.noarch
openstack-nova-14.0.2-7.el7ost.noarch
openstack-nova-common-14.0.2-7.el7ost.noarch




Steps to reproduce:
1. During upgrade OSP9->OSP10 - reach the step for upgrading computes.
2. Migrate VMs from one compute to any target using:
nova host-servers-migrate [hostname] fails 

3. Upgrade the empty compute
4. Then try to migrate vms from non upgraded compute to the upgraded one 

Result:

This resulted in error:
Grepping for errors the /var/log/nova/nova-compute.log file on a target compute:

2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server [req-5bbd4371-dd07-443c-884f-02937064bb8f 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server return [device for device in pci_devices if
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:21.693 31237 ERROR oslo_messaging.rpc.server
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Setting instance vm_state to ERROR
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] Traceback (most recent call last):
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] disk_info, image_meta)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] old_instance_type)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] self.force_reraise()
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] block_device_info, power_on)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] block_device_info=block_device_info)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] context)
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] return [device for device in pci_devices if
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef] TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:22.845 31237 ERROR nova.compute.manager [instance: a6a47318-dab5-4680-8bc3-be48c15568ef]
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server [req-10021f30-ac47-495d-83b6-2fe54d4ab647 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server return [device for device in pci_devices if
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 18:44:23.317 31237 ERROR oslo_messaging.rpc.server
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: e9797290-6333-485f-a1c4-965355b11874] Setting instance vm_state to ERROR
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] Traceback (most recent call last):
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] disk_info, image_meta)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] old_instance_type)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] self.force_reraise()
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] block_device_info, power_on)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] block_device_info=block_device_info)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] context)
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] if pci_devices is None:
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874] TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:19.571 31237 ERROR nova.compute.manager [instance: e9797290-6333-485f-a1c4-965355b11874]
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server [req-7ecfc3ee-2b17-4582-9788-f81a90de21e8 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server if pci_devices is None:
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:20.019 31237 ERROR oslo_messaging.rpc.server
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Setting instance vm_state to ERROR
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] Traceback (most recent call last):
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] disk_info, image_meta)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] old_instance_type)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] self.force_reraise()
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] block_device_info, power_on)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] block_device_info=block_device_info)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] context)
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] if pci_devices is None:
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4] TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:20.609 31237 ERROR nova.compute.manager [instance: 748520c6-6504-4c8e-9db1-cad2981715c4]
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server [req-81456147-61d3-4d1e-ac49-6becc999f201 929ce5e2752e4705bcc6d32e303230a9 2a99364bbb3c496290fcf71066bdfbfb - - -] Exception during message handling
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 145, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server migration.instance_uuid, exc_info=True)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 122, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3960, in finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3948, in finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server disk_info, image_meta)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3913, in _finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server old_instance_type)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3908, in _finish_resize
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7259, in finish_migration
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server block_device_info=block_device_info)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4652, in _get_guest_xml
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server context)
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4418, in _get_guest_config
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server pci_devs = pci_manager.get_instance_pci_devs(instance, 'all')
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/pci/manager.py", line 339, in get_instance_pci_devs
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server if pci_devices is None:
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2016-12-09 20:47:21.122 31237 ERROR oslo_messaging.rpc.server


Expected result:
Successful migration.


https://bugs.launchpad.net/nova/+bug/1648887
Comment 1 Sofer Athlan-Guyot 2016-12-09 21:24:21 UTC 
Adding launchpad and upstream gerrit.
Comment 3 Ofer Blaut 2016-12-14 08:33:44 UTC 
Is it also relate to https://bugs.launchpad.net/nova/+bug/1634282 ?
Comment 4 Jon Schlueter 2016-12-14 17:52:30 UTC 
adding stable/newton upstream backport to external trackers instead of master
Comment 10 Prasanth Anbalagan 2017-01-27 18:07:55 UTC 
Verified as follows,

********
VERSION
********

Before upgrade
===============

openstack-nova-compute.noarch        1:13.1.2-9.el7ost   @rhos-9.0-signed       

After upgrade
==============

openstack-nova-compute.noarch     1:14.0.3-1.el7ost     @rhelosp-10.0-puddle    

********
LOGS
********

Live-migration after upgrade of compute-1 to 10
===============================================

[heat-admin@compute-1 ~]$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------+
| ID                                   | Name | Status | Task State | Power State | Networks          |
+--------------------------------------+------+--------+------------+-------------+-------------------+
| a548bedc-26d9-492a-b1fd-34a51e6fa255 | vm1  | ACTIVE | -          | Running     | public=10.0.0.210 |
+--------------------------------------+------+--------+------------+-------------+-------------------+
[heat-admin@compute-1 ~]$ nova show vm1 | grep host
| OS-EXT-SRV-ATTR:host                 | compute-0.localdomain                                    |
| OS-EXT-SRV-ATTR:hostname             | vm1                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-0.localdomain                                    |
| hostId                               | fd25478cec5d1bb468981b78ad3dbd1cbf025103ba10fc7f12a8d99b |
| host_status                          | UP                                                       |
[heat-admin@compute-1 ~]$ 
[heat-admin@compute-1 ~]$ 
[heat-admin@compute-1 ~]$ nova live-migration --block-migrate vm1 compute-1.localdomain
[heat-admin@compute-1 ~]$ nova list
+--------------------------------------+------+-----------+------------+-------------+-------------------+
| ID                                   | Name | Status    | Task State | Power State | Networks          |
+--------------------------------------+------+-----------+------------+-------------+-------------------+
| a548bedc-26d9-492a-b1fd-34a51e6fa255 | vm1  | MIGRATING | migrating  | Running     | public=10.0.0.210 |
+--------------------------------------+------+-----------+------------+-------------+-------------------+

[heat-admin@compute-1 ~]$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------+
| ID                                   | Name | Status | Task State | Power State | Networks          |
+--------------------------------------+------+--------+------------+-------------+-------------------+
| a548bedc-26d9-492a-b1fd-34a51e6fa255 | vm1  | ACTIVE | -          | Running     | public=10.0.0.210 |
+--------------------------------------+------+--------+------------+-------------+-------------------+
[heat-admin@compute-1 ~]$ nova show vm1 | grep host
| OS-EXT-SRV-ATTR:host                 | compute-1.localdomain                                    |
| OS-EXT-SRV-ATTR:hostname             | vm1                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1.localdomain                                    |
| hostId                               | a7cbc82359561dde6dd4f70dc3ecf865666ae78b067db6641b7895b7 |
| host_status                          | UP                                                       |
[heat-admin@compute-1 ~]$
Comment 20 errata-xmlrpc 2017-02-23 21:14:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0319.html
Note You need to log in before you can comment on or make changes to this bug.