Bug 1447891 - LibvirtError happens when put instance from pause to active status
Summary: LibvirtError happens when put instance from pause to active status
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 8.0 (Liberty)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Sahid Ferdjaoui
QA Contact: awaugama
URL:
Whiteboard:
Depends On:
Blocks: 1448000 1448002
TreeView+ depends on / blocked
 
Reported: 2017-05-04 07:46 UTC by Meiyan Zheng
Modified: 2020-12-14 08:36 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1448000 (view as bug list)
Environment:
Last Closed: 2017-11-15 09:38:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1623089 0 None None None 2017-05-04 11:29:13 UTC

Description Meiyan Zheng 2017-05-04 07:46:33 UTC
Description of problem:

Following error happens when put instance from pause to active status:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
libvirtError: Requested operation is not valid: domain has assigned non-USB host devices
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Here are the traceback logs:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher [req-b2b1a797-2723-4cae-97e5-bc9acbfb9eeb 8d019c704cc34c2389913aa7532f606f 24aec973190e496f94690f0a1e7469fb - - -] Exception during message handling: Requested operation is not valid: domain has assigned non-USB host devices
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     payload)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4147, in suspend_instance
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     self.driver.suspend(context, instance)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2461, in suspend
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     guest.save_memory_state()
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 363, in save_memory_state
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     self._domain.managedSave(0)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1397, in managedSave
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainManagedSave() failed', dom=self)
2017-04-27 12:38:12.158 124385 ERROR oslo_messaging.rpc.dispatcher libvirtError: Requested operation is not valid: domain has assigned non-USB host devices
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++




Version-Release number of selected component (if applicable):

- OS: Red Hat Enterprise Linux Server release 7.2 (Maipo)
- Server: Dell Poweredge R630
- NIC Card: Intel 10G 2P X520 Adapter (SR-IOV)
- OpenStack: Red Hat OpenStack Platform 8
- openstack-nova-compute-12.0.4-8.el7ost.noarch  



How reproducible:

Steps to Reproduce:
1. Put instance from active to pause
2. Put instance from pause to active


Actual results:
Instance status becomes error

Expected results:
Instance can be activated normally



Additional info:

BTW, the sosreports can be downloaded from dropbox:
- Controller node: sosreport-VDM-I-MS-20170428080602.tar.xz
- Compute node:  sosreport-ca6.samsung.com-20170428080621.tar.xz

Case number: 01839181

From the traceback log seems match upstream bug: 
https://bugs.launchpad.net/nova/+bug/1623089

This bug is fixed in upstream version: openstack/nova 15.0.0.0b1

And looks like this patch isn't included in our latest nova-compute package:

/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py (OSP10)
...
    def _attach_sriov_ports(self, context, instance, guest, network_info=None):
        if network_info is None:
            network_info = instance.info_cache.network_info
        if network_info is None:
            return

        if self._has_sriov_port(network_info):
            for vif in network_info:
                if vif['vnic_type'] in network_model.VNIC_TYPES_SRIOV: <--- Changed point
                    cfg = self.vif_driver.get_config(instance,
                                                     vif,
                                                     instance.image_meta,
                                                     instance.flavor,
                                                     CONF.libvirt.virt_type,
                                                     self._host)



Expectation:
- Will this upstream bug fixed in next version? 
- Or do you have any backport plan on OSP8 for this upstream bug?

Comment 1 Sahid Ferdjaoui 2017-05-04 11:29:14 UTC
We can't really ensure that there are using SRIOV since we do not have domain XML but you have reported they are using a SRIOV card and the traceback looks really similar to the one reported against upstream. 

So I would say It's valid issue which can be backported for OSP8.

Comment 7 Sahid Ferdjaoui 2017-05-23 12:21:23 UTC
Looking at this one little bit deeper. The support of using vnic type direct-physical has been introduced by blueprint: sriov-pf-passthrough-neutron-port in RHOS9. So even if Neutron is permitting to create a port with such VNIC type, I'm not sure whether we can support it on RHOS8.

Comment 9 Lee Yarwood 2017-10-06 15:02:48 UTC
Closing with INSUFFICIENT_DATA, please feel free to reopen if you can provide n-cpu DEBUG logs *and* detailed steps to reproduce including the commands used to create the associated ports.

Comment 13 Sahid Ferdjaoui 2017-11-06 09:07:05 UTC
I suspect the issue related to libvirt so it's important to get the DEBUG information of Nova and libvird.

Comment 14 Sahid Ferdjaoui 2017-11-15 09:38:58 UTC
Please re-open with accurate logs if issue is still reproducible.


Note You need to log in before you can comment on or make changes to this bug.