Bug 2312890 - Client eviction fails with "no hosts match"
Summary: Client eviction fails with "no hosts match"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: CephFS
Version: 7.1
Hardware: All
OS: All
unspecified
urgent
Target Milestone: ---
: 7.1z2
Assignee: Venky Shankar
QA Contact: Hemanth Kumar
URL:
Whiteboard:
Depends On:
Blocks: 2315686
TreeView+ depends on / blocked
 
Reported: 2024-09-17 18:10 UTC by Goutham Pacha Ravi
Modified: 2025-03-30 12:29 UTC (History)
12 users (show)

Fixed In Version: ceph-18.2.1-247.el9cp
Doc Type: Bug Fix
Doc Text:
.Ceph MDS evict command works as expected Previously, the Ceph Metadata Server (MDS) incorrectly rejected by evicting invalid sessions with an error code. As a result, layered products that relied on the Ceph MDS `evict` command gave an exception. With this fix, cases of invalid session IDs are handled as no-operation, and users or products relying on Ceph MDS work as expected.
Clone Of:
: 2315686 (view as bug list)
Environment:
Last Closed: 2024-11-07 14:39:02 UTC
Embargoed:
dwalveka: needinfo-
dwalveka: needinfo-
dwalveka: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 68132 0 None None None 2024-09-18 16:31:38 UTC
Red Hat Issue Tracker RHCEPH-9809 0 None None None 2024-09-18 16:34:43 UTC
Red Hat Issue Tracker RHCEPH-9815 0 None None None 2024-09-19 17:07:27 UTC
Red Hat Issue Tracker RHCEPH-9847 0 None None None 2024-09-17 18:10:53 UTC
Red Hat Product Errata RHBA-2024:9010 0 None None None 2024-11-07 14:39:11 UTC

Description Goutham Pacha Ravi 2024-09-17 18:10:35 UTC 
Description of problem:

OpenStack Manila provisions CephFS subvolumes as "shares" to OpenStack tenants. CephX users can be granted access to read/write into these shares. Manila uses the following Ceph Volume APIs to create and delete access rules:


Authorize:

ceph_argparse.json_command - rados_client=<rados.Rados object at 0x7f0ed5d95fa0>, target=('mon-mgr',), prefix='fs subvolume authorize', argdict={'vol_name': 'cephfs', 'sub_name': '08748d00-6327-43de-8f9d-d25234dd5fc6', 'auth_id': 'eve49', 'tenant_id': '84cf269d7d844e658b4c5024db961709', 'access_level': 'rw', 'format': 'json'}, inbuf=b'', timeout=10.


Deauthorize:

ceph_argparse.json_command - rados_client=<rados.Rados object at 0x7f0ed5d95fa0>, target=('mon-mgr',), prefix='fs subvolume deauthorize', argdict={'vol_name': 'cephfs', 'sub_name': '08748d00-6327-43de-8f9d-d25234dd5fc6', 'auth_id': 'eve49', 'format': 'json'}, inbuf=b'', timeout=10.

Evict:

ceph_argparse.json_command - rados_client=<rados.Rados object at 0x7f0ed5d95fa0>, target=('mon-mgr',), prefix='fs subvolume evict', argdict={'vol_name': 'cephfs', 'sub_name': '08748d00-6327-43de-8f9d-d25234dd5fc6', 'auth_id': 'eve49', 'format': 'json'}, inbuf=b'', timeout=10.



These interactions work fine with the "7-424" tag of the RHCS container image on https://catalog.redhat.com/software/containers/rhceph/rhceph-7-rhel9/64c1382d898617ef712459be.

However, the "evict" API fails on "7-430" with the following traceback:



024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server [None req-ef8558bb-d0d7-4db2-979b-cf45ee092bd1 fcae00ce5a5643a39abd8c71ce6d9ee5 84cf269d7d844e658b4c5024db961709 - - - -] Exception during message handling: manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume evict, argdict={'vol_name': 'cephfs', 'sub_name': '08748d00-6327-43de-8f9d-d25234dd5fc6', 'auth_id': 'eve49', 'format': 'json'} - exception message: [errno -22] Failed to evict client with auth_name=eve49, client_metadata.root=/volumes/_nogroup/08748d00-6327-43de-8f9d-d25234dd5fc6/5ad7d3bc-1ae1-4279-bc2a-4c28cb522718 from mds 0/14451: Error -3 ("no hosts match") while Sending evict to mds.14451.
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 210, in rados_command
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     raise rados.Error(outs, ret)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server rados.Error: [errno -22] Failed to evict client with auth_name=eve49, client_metadata.root=/volumes/_nogroup/08748d00-6327-43de-8f9d-d25234dd5fc6/5ad7d3bc-1ae1-4279-bc2a-4c28cb522718 from mds 0/14451: Error -3 ("no hosts match") while Sending evict to mds.14451
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server 
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server 
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 226, in wrapped
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/utils.py", line 481, in wrapper
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 3496, in delete_share_instance
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     self.message_api.create(
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     raise self.value
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 3474, in delete_share_instance
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     self.access_helper.update_access_rules(
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/access.py", line 299, in update_access_rules
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     self._update_access_rules(context, share_instance_id,
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/access.py", line 336, in _update_access_rules
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     driver_rule_updates = self._update_rules_through_share_driver(
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/access.py", line 401, in _update_rules_through_share_driver
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     driver_rule_updates = self.driver.update_access(
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 546, in update_access
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     return self.protocol_helper.update_access(
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 1019, in update_access
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     self._deny_access(context, share, rule)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 936, in _deny_access
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     rados_command(self.rados_client, "fs subvolume evict", argdict)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 222, in rados_command
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server     raise exception.ShareBackendException(msg)
2024-09-12 12:53:25.892 16 ERROR oslo_messaging.rpc.server manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume evict, argdict={'vol_name': 'cephfs', 'sub_name': '08748d00-6327-43de-8f9d-d25234dd5fc6', 'auth_id': 'eve49', 'format': 'json'} - exception message: [errno -22] Failed to evict client with auth_name=eve49, client_metadata.root=/volumes/_nogroup/08748d00-6327-43de-8f9d-d25234dd5fc6/5ad7d3bc-1ae1-4279-bc2a-4c28cb522718 from mds 0/14451: Error -3 ("no hosts match") while Sending evict to mds.14451.
Comment 30 errata-xmlrpc 2024-11-07 14:39:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:9010
Note You need to log in before you can comment on or make changes to this bug.