Bug 1462259 - Trying to allow/deny access to a manila share fails with CephFS Native
Trying to allow/deny access to a manila share fails with CephFS Native
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-manila (Show other bugs)
11.0 (Ocata)
Unspecified Unspecified
medium Severity medium
: z2
: 11.0 (Ocata)
Assigned To: Tom Barron
Dustin Schoenbrun
Don Domingo
: Triaged, ZStream
Depends On:
Blocks: 1462527
  Show dependency treegraph
 
Reported: 2017-06-16 10:23 EDT by Raissa Sarmento
Modified: 2017-09-13 17:50 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1462527 (view as bug list)
Environment:
Last Closed: 2017-09-13 17:50:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Raissa Sarmento 2017-06-16 10:23:12 EDT
Description of problem:

If one tries to allow access to a user and this user's access_key is dirty somehow, manila will show the share access status as "error" and the share.log will show:

------------------------------------

2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server [req-a5b409f8-4d1f-487a-a887-3dd6e000c840 05e228f17d8b426b9071b1ee0857b814 c8efa291fa31410fb999251c4c66d5af - - -] Exception during message handling
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 167, in wrapped
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 519, in wrapper
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3073, in update_access
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 280, in update_access_rules
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 319, in _update_access_rules
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 380, in _update_rules_through_share_driver
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server=share_server
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 292, in update_access
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     access_key = self._allow_access(context, share, rule)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 241, in _allow_access
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     tenant_id=share['project_id'])
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 973, in authorize
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     key = self._authorize_volume(volume_path, auth_id, readonly)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1009, in _authorize_volume
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     key = self._authorize_ceph(volume_path, auth_id, readonly)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1086, in _authorize_ceph
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     'mon', cap['caps'].get('mon')]
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1275, in _rados_command
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     raise rados.Error(outs)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server Error: Can't handle arrays of non-strings

------------------------------------

Then, when trying to deny access to the user, nothing happens, but share.log show the following error: 

------------------------------------

2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server [req-16ac61ab-1c5d-4cab-b685-e4eb030380e1 05e228f17d8b426b9071b1ee0857b814 c8efa291fa31410fb999251c4c66d5af - - -] Exception during message handling
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 167, in wrapped
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 519, in wrapper
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3073, in update_access
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 280, in update_access_rules
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 319, in _update_access_rules
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 380, in _update_rules_through_share_driver
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server=share_server
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 296, in update_access
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     self._deny_access(context, share, rule)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 253, in _deny_access
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     access['access_to'])
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1133, in deauthorize
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     self._recover_auth_meta(auth_id, auth_meta)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server TypeError: _recover_auth_meta() takes exactly 2 arguments (3 given)

------------------------------------


Version-Release number of selected component (if applicable):

puppet-manila-10.3.0-1.el7ost.noarch
openstack-manila-share-4.0.0-4.el7ost.noarch
python-manilaclient-1.14.0-1.el7ost.noarch
python-manila-4.0.0-4.el7ost.noarch
openstack-manila-4.0.0-4.el7ost.noarch
openstack-manila-ui-2.7.1-2.el7ost.noarch

Most relevant to the problem:
python-cephfs-10.2.5-37.el7cp.x86_64


Steps to Reproduce:
1. Create a user using the following command (note that alice could be any other user):

ceph --name=client.manila --keyring=/etc/ceph/ceph.client.manila.keyring auth get-or-create client.alice -o alice.keyring

2. Try to allow that user to have access to the share:

manila access-allow <share_name> cephx alice 

3. Run 'manila access-list <share>' and see that the share is in 'error' state

4. Try to run 'manila access-deny <share> <access-id>'


Actual results:

Share access in error state and error trace in /var/log/manila/share.log (in the node that runs manila share service)

Expected results:

Access removed successfully

---

Additional info:

The solution for this problem was addressed 8 months ago and is included in python-cephfs-10.2.7-27.el7cp.x86_64.rpm - we should make sure RHOS11 includes the package with the bugfix, or maybe backport the fix.
Comment 2 Red Hat Bugzilla Rules Engine 2017-06-19 08:32:22 EDT
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 4 Brett Niver 2017-08-09 11:37:58 EDT
@Tom, we need to move this to either 12 or 13 if there's any CephFS work required, correct?
Comment 5 Dustin Schoenbrun 2017-08-09 11:46:21 EDT
This should be easy to verify with OSP-11 as the fix should be included in one of the latest builds. I have an environment set up to see if the new version was installed with the latest OSP-11 puddle and will report back accordingly.
Comment 6 Dustin Schoenbrun 2017-08-09 17:39:14 EDT
Using the latest OSP-11 puddle (2017-08-02.1), I was able to verify that the correct version of the python-cephfs package was installed on the controller and ceph nodes in the overcloud and that I was able to successfully grant access to a cephx ID as well as deny access to the same without issue. Logs are below for the granting and denying of access:

2017-08-09 21:33:43.974 195420 INFO ceph_volume_client [req-74209b11-13f6-4eac-8333-e31594e65589 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] create_volume: /volumes/_nogroup/8ef47735-efe1-4783-890c-a52bb4221d3f
2017-08-09 21:33:44.815 195420 INFO ceph_volume_client [req-74209b11-13f6-4eac-8333-e31594e65589 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] create_volume: None/8ef47735-efe1-4783-890c-a52bb4221d3f, using rados namespace fsvolumens_8ef47735-efe1-4783-890c-a52bb4221d3f to isolate data.
2017-08-09 21:33:44.838 195420 INFO ceph_volume_client [req-74209b11-13f6-4eac-8333-e31594e65589 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] get_mon_addrs
2017-08-09 21:33:44.902 195420 INFO manila.share.drivers.cephfs.cephfs_native [req-74209b11-13f6-4eac-8333-e31594e65589 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] Calculated export location for share 8ef47735-efe1-4783-890c-a52bb4221d3f: 172.17.3.11:6789,172.17.3.14:6789,172.17.3.23:6789:/volumes/_nogroup/8ef47735-efe1-4783-890c-a52bb4221d3f
2017-08-09 21:33:45.188 195420 INFO manila.share.manager [req-74209b11-13f6-4eac-8333-e31594e65589 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] Share instance 8ef47735-efe1-4783-890c-a52bb4221d3f created successfully.
2017-08-09 21:34:26.047 195420 INFO manila.share.manager [req-9572830f-4fdd-4828-96b9-c56d54fa5136 - - - - -] Updating share status
2017-08-09 21:34:44.670 195420 INFO manila.share.access [req-9d315340-a32e-4128-8aaa-e14567eb158a 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] Access rules were successfully modified for share instance 8ef47735-efe1-4783-890c-a52bb4221d3f belonging to share 4cc1e0de-6747-42dc-b58c-4af3493531a7.
2017-08-09 21:35:23.295 195420 INFO ceph_volume_client [req-f3357769-3d14-46f1-b373-4396781ee56f 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] evict clients with auth_name=alice, client_metadata.root=/volumes/_nogroup/8ef47735-efe1-4783-890c-a52bb4221d3f
2017-08-09 21:35:23.507 195420 INFO ceph_volume_client [req-f3357769-3d14-46f1-b373-4396781ee56f 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] evict: joined all
2017-08-09 21:35:23.783 195420 INFO manila.share.access [req-f3357769-3d14-46f1-b373-4396781ee56f 06ad0f74a2c74f6cb652a6ca593a7d0f 2112b6124a0941af8bf4e0806c681f31 - - -] Access rules were successfully modified for share instance 8ef47735-efe1-4783-890c-a52bb4221d3f belonging to share 4cc1e0de-6747-42dc-b58c-4af3493531a7.
Comment 7 Tom Barron 2017-08-09 20:58:22 EDT
(In reply to Brett Niver from comment #4)
> @Tom, we need to move this to either 12 or 13 if there's any CephFS work
> required, correct?

Dustin has verified that we have the fix in 11 so we don't need to move it out to a later release.
Comment 10 errata-xmlrpc 2017-09-13 17:50:42 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2722

Note You need to log in before you can comment on or make changes to this bug.