Description of problem:

We can't seem to be able to evacuate instances from a failed compute node using shared storage. We are using Ceph Ephemeral RBD as the storage medium.

Version-Release number of selected component (if applicable):

Havana, Icehouse, Juno

How reproducible:
Always

Steps to Reproduce:

nova evacuate --on-shared-storage 6e2081ec-2723-43c7-a730-488bb863674c node-24
or
POST to http://ip-address:port/v2/tenant_id/servers/server_id/action with
{"evacuate":{"host":"node-24","onSharedStorage":1}}

Here is what shows up in the logs:

180>Jul 10 20:36:48 node-24 nova-nova.compute.manager AUDIT: Rebuilding instance
<179>Jul 10 20:36:48 node-24 nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5554, in _error_out_instance_on_exception
    yield
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2434, in rebuild_instance
    _("Invalid state of instance files on shared"
InvalidSharedStorage: Invalid state of instance files on shared storage
<179>Jul 10 20:36:49 node-24 nova-oslo.messaging.rpc.dispatcher ERROR: Exception during message handling: Invalid state of instance files on shared storage
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
    payload)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
    pass
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
    function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
    e, sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2434, in rebuild_instance
    _("Invalid state of instance files on shared"
InvalidSharedStorage: Invalid state of instance files on shared storage