Bug 1257719 - Nova resize fails with ceph as default ephemeral storage
Summary: Nova resize fails with ceph as default ephemeral storage
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Eoghan Glynn
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-08-27 18:53 UTC by David Paterson
Modified: 2019-09-09 17:03 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-28 14:14:42 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description David Paterson 2015-08-27 18:53:05 UTC 
Description of problem: When attempting to resize an instance to a different flavor nova fails.


Version-Release number of selected component (if applicable):
Nova packages are version: 2015.1.0-17.el7ost

How reproducible: Every time


Steps to Reproduce:
1. Log into Horizon
2. Create an instance
3. Resize instance and select a different flavor

Actual results: Error is logged


Expected results: Instance is resized to new flavor


Additional info:
Configuration is using ceph for ephemeral storage.


Stack trace from compute node nova-compute.log:

2015-08-27 17:48:39.310 119972 ERROR oslo_messaging.rpc.dispatcher [req-aea9b128-1d33-4b9b-95c4-ccd48967bff2 5927e405d1c247de94453a0d11806b0a       ae6bb21be1ce46bca82bd08d8aa8748e - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh 192.168.120.49 mkdir -p /var/lib/nova/instances/ec127575-7f69-4012-a932-c20dd5dfeb3d
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",       line 142, in _dispatch_and_reply
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",       line 186, in _dispatch
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",       line 130, in _do_dispatch
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 675      5, in resize_instance
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     clean_shutdown=clean_shutdown)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wr      apped
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wr      apped
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 327      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 377      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 286      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 269      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,       in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343      , in decorated_function
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 401      9, in resize_instance
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     self.instance_events.clear_events_for_instance(instance)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     self.gen.throw(type, value, traceback)
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 638      9, in _error_out_instance_on_exception
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected erro      r while running command.
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Command: ssh 192.168.120.49 mkdir -p /var/lib/nova/instances/ec127575-7f69-      4012-a932-c20dd5dfeb3d
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'
2015-08-27 17:48:39.310 119972 TRACE oslo_messaging.rpc.dispatcher
2015-08-27 17:53:25.971 119972 WARNING nova.virt.libvirt.imagecache [req-6ee6edb5-d1ad-4a28-9e94-78fdf55e2f5e - - - - -] Unknown base file: /v      ar/lib/nova/instances/_base/0fff935e5fa138a2c794064ffc74ac585dc5bcce
Comment 5 Daniel Berrangé 2015-08-28 14:14:42 UTC 
Nova uses SSH to access to the remote machine and SSH is throwing an error:

 Stderr: u'Host key verification failed.\r\n'

this indicates a host OS configuration problem. Possibly you changed the host key of the remote host, or perhaps the host has never been logged into at all before.

So I don't think this is a really classed as a Nova bug.
Note You need to log in before you can comment on or make changes to this bug.