Bug 1257719

Summary:	Nova resize fails with ceph as default ephemeral storage
Product:	Red Hat OpenStack	Reporter:	David Paterson <david_paterson>
Component:	openstack-nova	Assignee:	Eoghan Glynn <eglynn>
Status:	CLOSED NOTABUG	QA Contact:	nlevinki <nlevinki>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	arkady_kanevsky, berrange, cdevine, christopher_dearborn, dasmith, david_paterson, eglynn, John_walsh, kbader, kchamart, kurt_hey, morazi, ndipanov, pbrady, rajini.karthik, randy_perryman, rsussman, sbauza, sferdjao, sgordon, sreichar, vromanso, wayne_allen, yeylon
Target Milestone:	---
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-08-28 14:14:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Paterson 2015-08-27 18:53:05 UTC

Description of problem: When attempting to resize an instance to a different flavor nova fails.


Version-Release number of selected component (if applicable):
Nova packages are version: 2015.1.0-17.el7ost

How reproducible: Every time


Steps to Reproduce:
1. Log into Horizon
2. Create an instance
3. Resize instance and select a different flavor

Actual results: Error is logged


Expected results: Instance is resized to new flavor


Additional info:
Configuration is using ceph for ephemeral storage.


Stack trace from compute node nova-compute.log:

2015-08-27 17:48:39.310 119972 ERROR oslo_messaging.rpc.dispatcher [req-aea9b128-1d33-4b9b-95c4-ccd48967bff2 5927e405d1c247de94453a0d11806b0a       ae6bb21be1ce46bca82bd08d8aa8748e - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh 192.168.120.49 mkdir -p /var/lib/nova/instances/ec127575-7f69-4012-a932-c20dd5dfeb3d
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",       line 142, in _dispatch_and_reply
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",       line 186, in _dispatch
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",       line 130, in _do_dispatch
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 675      5, in resize_instance
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     clean_shutdown=clean_shutdown)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wr      apped
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wr      apped
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 327      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 377      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 286      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 269      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 401      9, in resize_instance
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     self.instance_events.clear_events_for_instance(instance)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     self.gen.throw(type, value, traceback)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 638      9, in _error_out_instance_on_exception
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected erro      r while running command.
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Command: ssh 192.168.120.49 mkdir -p /var/lib/nova/instances/ec127575-7f69-      4012-a932-c20dd5dfeb3d
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher
2015-08-27 17:53:25.971 119972 WARNING nova.virt.libvirt.imagecache [req-6ee6edb5-d1ad-4a28-9e94-78fdf55e2f5e - - - - -] Unknown base file: /v      ar/lib/nova/instances/_base/0fff935e5fa138a2c794064ffc74ac585dc5bcce

Comment 5 Daniel Berrangé 2015-08-28 14:14:42 UTC

Nova uses SSH to access to the remote machine and SSH is throwing an error:

 Stderr: u'Host key verification failed.\r\n'

this indicates a host OS configuration problem. Possibly you changed the host key of the remote host, or perhaps the host has never been logged into at all before.

So I don't think this is a really classed as a Nova bug.