Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1436593

Summary:	Not able to create snapshot of dpdk based instance.
Product:	Red Hat OpenStack	Reporter:	VIKRANT <vaggarwa>
Component:	openstack-nova	Assignee:	Eoghan Glynn <eglynn>
Status:	CLOSED DUPLICATE	QA Contact:	Prasanth Anbalagan <panbalag>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	10.0 (Newton)	CC:	berrange, dasmith, eglynn, kchamart, sbauza, sferdjao, sgordon, srevivo, stephenfin, vaggarwa, vromanso
Target Milestone:	---	Flags:	vaggarwa: needinfo+
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-07 12:59:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description VIKRANT 2017-03-28 09:13:35 UTC

Description of problem:

Not able to create snapshot of dpdk based instance. 

Version-Release number of selected component (if applicable):
RHEL OSP 10
OVS-DPDK 2.5

How reproducible:
Everytime. 

Steps to Reproduce:
1. Spawn an instance using dpdk interface. 
2. Try to take the snapshot of instance:
$ nova image-create 0c9e07ff-ef36-4198-8def-8a66f3d8c63c rhel7-image-002
3. It's getting failed with call trace in nova-compute.log file of compute node. 

Actual results:
Snapshot operation is getting failed. 

Expected results:
Snapshot operation should get completed. 

Additional info:

Error reported in log file :

~~~
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server [req-f815e4d0-1555-459d-9e33-6a591e003e6a fb779c2a36a34a95b04571ecad682431 b08633e159af4c8c8757fb31e7ed41e4 - - -] Exception during message handling
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 242, in decorated_function
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     instance=instance)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 232, in decorated_function
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     *args, **kwargs)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3038, in snapshot_instance
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     task_states.IMAGE_SNAPSHOT)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3068, in _snapshot_instance
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     update_task_state)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1470, in snapshot
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     instance)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1563, in _prepare_domain_for_snapshot
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     self.suspend(context, instance)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2445, in suspend
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     guest.save_memory_state()
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 425, in save_memory_state
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     self._domain.managedSave(0)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1417, in managedSave
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainManagedSave() failed', dom=self)
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command 'migrate': Migration disabled: vhost-user backend lacks VHOST_USER_PROTOCOL_F_LOG_SHMFD feature.
2017-03-28 08:53:34.075 49980 ERROR oslo_messaging.rpc.server 
~~~

Comment 1 Stephen Finucane 2017-04-07 11:33:35 UTC

This appears to be an issue with the DPDK version used. Support for vhost-user live migration was enabled in DPDK commit d639996a [1], which is available in DPDK 16.04 and up:

    $ cd dpdk
    $ git tag --contains d639996a
    v16.04
    v16.04-rc1
    v16.04-rc2
    v16.04-rc3
    v16.04-rc4
    ...

Could you confirm that DPDK is one of these supported versions? If not, a newer DPDK version will be required else this feature will not be possible.

[1] http://dpdk.org/browse/dpdk/commit/?id=d639996a

Comment 2 Sahid Ferdjaoui 2017-04-07 11:49:27 UTC

Seems related to:

   https://bugzilla.redhat.com/show_bug.cgi?id=1436281

My thinking is a problem related to the guest kernel, which version of RHEL are you using?

Comment 3 Stephen Finucane 2017-04-07 12:55:21 UTC

(In reply to Sahid Ferdjaoui from comment #2)
> Seems related to:
> 
>    https://bugzilla.redhat.com/show_bug.cgi?id=1436281
> 
> My thinking is a problem related to the guest kernel, which version of RHEL
> are you using?

Kernel networking isn't being used here though - it's a DPDK interface. However, the bug report you link to does help - bz1335865 states that you *do* need OVS 2.6. Seeing as the CU is using 2.5, this would explain the issue. They will need to update if they want to use this feature.

Closing this as a duplicate of the aforementioned bug.

Comment 4 Stephen Finucane 2017-04-07 12:59:57 UTC


*** This bug has been marked as a duplicate of bug 1335865 ***

Comment 5 awaugama 2017-09-07 19:04:12 UTC

Dup -- QE will decide about automating the original