Description of problem: Following error happens when put instance from pause to active status: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ libvirtError: Requested operation is not valid: domain has assigned non-USB host devices +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Here are the traceback logs: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher [req-b2b1a797-2723-4cae-97e5-bc9acbfb9eeb 8d019c704cc34c2389913aa7532f606f 24aec973190e496f94690f0a1e7469fb - - -] Exception during message handling: Requested operation is not valid: domain has assigned non-USB host devices 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher payload) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4147, in suspend_instance 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher self.driver.suspend(context, instance) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2461, in suspend 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher guest.save_memory_state() 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 363, in save_memory_state 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher self._domain.managedSave(0) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher six.reraise(c, e, tb) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1397, in managedSave 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainManagedSave() failed', dom=self) 2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher libvirtError: Requested operation is not valid: domain has assigned non-USB host devices +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Version-Release number of selected component (if applicable): - OS: Red Hat Enterprise Linux Server release 7.2 (Maipo) - Server: Dell Poweredge R630 - NIC Card: Intel 10G 2P X520 Adapter (SR-IOV) - OpenStack: Red Hat OpenStack Platform 8 - openstack-nova-compute-12.0.4-8.el7ost.noarch How reproducible: Steps to Reproduce: 1. Put instance from active to pause 2. Put instance from pause to active Actual results: Instance status becomes error Expected results: Instance can be activated normally Additional info: BTW, the sosreports can be downloaded from dropbox: - Controller node: sosreport-VDM-I-MS-20170428080602.tar.xz - Compute node: sosreport-ca6.samsung.com-20170428080621.tar.xz Case number: 01839181 From the traceback log seems match upstream bug: https://bugs.launchpad.net/nova/+bug/1623089 This bug is fixed in upstream version: openstack/nova 15.0.0.0b1 And looks like this patch isn't included in our latest nova-compute package: /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py (OSP10) ... def _attach_sriov_ports(self, context, instance, guest, network_info=None): if network_info is None: network_info = instance.info_cache.network_info if network_info is None: return if self._has_sriov_port(network_info): for vif in network_info: if vif['vnic_type'] in network_model.VNIC_TYPES_SRIOV: <--- Changed point cfg = self.vif_driver.get_config(instance, vif, instance.image_meta, instance.flavor, CONF.libvirt.virt_type, self._host) Expectation: - Will this upstream bug fixed in next version? - Or do you have any backport plan on OSP8 for this upstream bug?
We can't really ensure that there are using SRIOV since we do not have domain XML but you have reported they are using a SRIOV card and the traceback looks really similar to the one reported against upstream. So I would say It's valid issue which can be backported for OSP8.
Looking at this one little bit deeper. The support of using vnic type direct-physical has been introduced by blueprint: sriov-pf-passthrough-neutron-port in RHOS9. So even if Neutron is permitting to create a port with such VNIC type, I'm not sure whether we can support it on RHOS8.
Closing with INSUFFICIENT_DATA, please feel free to reopen if you can provide n-cpu DEBUG logs *and* detailed steps to reproduce including the commands used to create the associated ports.
I suspect the issue related to libvirt so it's important to get the DEBUG information of Nova and libvird.
Please re-open with accurate logs if issue is still reproducible.