Bug 1448002

+++ This bug was initially created as a clone of Bug #1448000 +++

+++ This bug was initially created as a clone of Bug #1447891 +++

Description of problem:

Following error happens when put instance from pause to active status:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
libvirtError: Requested operation is not valid: domain has assigned non-USB host devices
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Here are the traceback logs:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher [req-b2b1a797-2723-4cae-97e5-bc9acbfb9eeb 8d019c704cc34c2389913aa7532f606f 24aec973190e496f94690f0a1e7469fb - - -] Exception during message handling: Requested operation is not valid: domain has assigned non-USB host devices
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     payload)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4147, in suspend_instance
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     self.driver.suspend(context, instance)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2461, in suspend
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     guest.save_memory_state()
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 363, in save_memory_state
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     self._domain.managedSave(0)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1397, in managedSave
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainManagedSave() failed', dom=self)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher libvirtError: Requested operation is not valid: domain has assigned non-USB host devices
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++




Version-Release number of selected component (if applicable):

- OS: Red Hat Enterprise Linux Server release 7.2 (Maipo)
- Server: Dell Poweredge R630
- NIC Card: Intel 10G 2P X520 Adapter (SR-IOV)
- OpenStack: Red Hat OpenStack Platform 8
- openstack-nova-compute-12.0.4-8.el7ost.noarch  



How reproducible:

Steps to Reproduce:
1. Put instance from active to pause
2. Put instance from pause to active


Actual results:
Instance status becomes error

Expected results:
Instance can be activated normally



Additional info:

BTW, the sosreports can be downloaded from dropbox:
- Controller node: sosreport-VDM-I-MS-20170428080602.tar.xz
- Compute node:  sosreport-ca6.samsung.com-20170428080621.tar.xz

Case number: 01839181

From the traceback log seems match upstream bug: 
https://bugs.launchpad.net/nova/+bug/1623089

This bug is fixed in upstream version: openstack/nova 15.0.0.0b1

And looks like this patch isn't included in our latest nova-compute package:

/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py (OSP10)
...
    def _attach_sriov_ports(self, context, instance, guest, network_info=None):
        if network_info is None:
            network_info = instance.info_cache.network_info
        if network_info is None:
            return

        if self._has_sriov_port(network_info):
            for vif in network_info:
                if vif['vnic_type'] in network_model.VNIC_TYPES_SRIOV: <--- Changed point
                    cfg = self.vif_driver.get_config(instance,
                                                     vif,
                                                     instance.image_meta,
                                                     instance.flavor,
                                                     CONF.libvirt.virt_type,
                                                     self._host)



Expectation:
- Will this upstream bug fixed in next version? 
- Or do you have any backport plan on OSP8 for this upstream bug?

--- Additional comment from Sahid Ferdjaoui on 2017-05-04 07:29:14 EDT ---

We can't really ensure that there are using SRIOV since we do not have domain XML but you have reported they are using a SRIOV card and the traceback looks really similar to the one reported against upstream. 

So I would say It's valid issue which can be backported for OSP8.