This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 1814769 - [Manila] force-delete doesn't seem to work if host underlying share is unavailable
Summary: [Manila] force-delete doesn't seem to work if host underlying share is unavai...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-manila
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: OpenStack Manila Bugzilla Bot
QA Contact: vhariria
Erin Peterson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-18 16:11 UTC by Victoria Martinez de la Cruz
Modified: 2024-01-19 06:42 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-01-18 21:45:43 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1867030 0 None None None 2022-05-04 16:04:06 UTC
Red Hat Issue Tracker OSP-12583 0 None None None 2022-02-09 16:56:22 UTC
Red Hat Issue Tracker OSP-31213 0 None None None 2024-01-18 21:47:33 UTC
Red Hat Issue Tracker   OSPRH-3887 0 None None None 2024-01-19 06:42:18 UTC

Internal Links: 1812281

Description Victoria Martinez de la Cruz 2020-03-18 16:11:16 UTC 
The ability to force delete a share [1][2] was provided as an admin API [3] to remove shares from manila despite backend failures. However, this API does not account for times when the backend share service is not responding. For example:

$ manila service-list --binary manila-share
+----+--------------+------------------+---------+---------+-------+----------------------------+
| Id | Binary | Host | Zone | Status | State | Updated_at |
+----+--------------+------------------+---------+---------+-------+----------------------------+
| 8 | manila-share | ostk-train@beta | chicago | enabled | down | 2020-03-11T17:15:58.000000 |
| 9 | manila-share | ostk-train@delta | chicago | enabled | down | 2020-03-11T17:21:15.000000 |
| 10 | manila-share | ostk-train@gamma | dallas | enabled | down | 2020-03-11T17:15:49.000000 |
| 11 | manila-share | ostk-train@alpha | dallas | enabled | down | 2020-03-11T17:15:54.000000 |
+----+--------------+------------------+---------+---------+-------+----------------------------+

In this situation, if force deleting a share on a backend reporting as "down" leaves the share in "deleting" state forever.

The error in the API logs suggests that the host binary was not found. This error isn't being handled:

ERROR manila.api.middleware.fault [None req-c3c161dc-ddc3-478f-a538-6f36ecd98bb6 None None]
Caught error: Could not find binary ostk-train@delta on host manila-share.: manila.exception.HostBinaryNotFound: Could not find binary ostk-train@delta on host man
ila-share.
ERROR manila.api.middleware.fault Traceback (most recent call last):
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/middleware/fault.py",
 line 77, in __call__
ERROR manila.api.middleware.fault return req.get_response(self.application)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/oslo_middle
ware/base.py", line 130, in __call__
ERROR manila.api.middleware.fault response = req.get_response(self.application)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/keystonemid
dleware/auth_token/__init__.py", line 341, in __call__
ERROR manila.api.middleware.fault response = req.get_response(self._app)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/routes/midd
leware.py", line 141, in __call__
ERROR manila.api.middleware.fault response = self.app(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", l
ine 766, in __call__
ERROR manila.api.middleware.fault content_type, body, accept)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", l
ine 830, in _process_stack
ERROR manila.api.middleware.fault action_result = self.dispatch(meth, request, action_ar
gs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 925, in dispatch
ERROR manila.api.middleware.fault return method(req=request, **action_args)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1068, in version_select
ERROR manila.api.middleware.fault return func.func(self, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/v2/services.py", line 125, in update
ERROR manila.api.middleware.fault return self._update(req, id, body)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1162, in wrapper
ERROR manila.api.middleware.fault return f(self, req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/v2/services.py", line 89, in _update
ERROR manila.api.middleware.fault svc = db.service_get_by_args(context, data['host'], data['binary'])
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/api.py", line 119, in service_get_by_args
ERROR manila.api.middleware.fault return IMPL.service_get_by_args(context, host, binary)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 152, in wrapper
ERROR manila.api.middleware.fault return f(*args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 462, in service_get_by_args
ERROR manila.api.middleware.fault raise exception.HostBinaryNotFound(host=host, binary=binary)
ERROR manila.api.middleware.fault manila.exception.HostBinaryNotFound: Could not find binary ostk-train@delta on host manila-share.
ERROR manila.api.middleware.fault
Comment 3 Chuck Copello 2020-06-30 13:19:05 UTC 
Changed Doc Contact to default email list.
Comment 4 Yaniv Kaul 2022-05-11 13:32:49 UTC 
It was not handled in 2 years and no customer cases - do we plan to fix this? For 17.0?
Comment 5 Goutham Pacha Ravi 2022-05-11 19:05:28 UTC 
(In reply to Yaniv Kaul from comment #4)
> It was not handled in 2 years and no customer cases - do we plan to fix
> this? For 17.0?

Yes, this bug was a clone of https://bugzilla.redhat.com/show_bug.cgi?id=1812281 and the customer that needed this was provided a workaround. The heat on this RFE is very low since its quite rare to need to delete resources directly from the database without consulting the backend storage system, which, in this use case is unavailable. 

We recently decided to pursue a different direction with "unmanage" workflows. We'll add an optional "--skip-host-check" parameter to share and snapshot unmanage APIs to support removing something from the database without forwarding the request to the backend storage driver. I'll retarget this to OSP 18.0 since it involves an API change (and hence cannot be backported to OSP 17).
Note You need to log in before you can comment on or make changes to this bug.