Bug 1462527

Summary:	Trying to allow/deny access to a manila share fails with CephFS Native
Product:	Red Hat OpenStack	Reporter:	Tom Barron <tbarron>
Component:	openstack-manila	Assignee:	Tom Barron <tbarron>
Status:	CLOSED ERRATA	QA Contact:	Dustin Schoenbrun <dschoenb>
Severity:	medium	Docs Contact:	Don Domingo <ddomingo>
Priority:	medium
Version:	10.0 (Newton)	CC:	ddomingo, dschoenb, jjoyce, rdearauj, slinaber
Target Milestone:	z4	Keywords:	TestOnly, Triaged, ZStream
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-cephfs-10.2.7-27.el7cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1462259	Environment:
Last Closed:	2017-09-06 17:06:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1462259
Bug Blocks:

Description Tom Barron 2017-06-18 13:43:44 UTC

+++ This bug was initially created as a clone of Bug #1462259 +++

Description of problem:

If one tries to allow access to a user and this user's access_key is dirty somehow, manila will show the share access status as "error" and the share.log will show:

------------------------------------

2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server [req-a5b409f8-4d1f-487a-a887-3dd6e000c840 05e228f17d8b426b9071b1ee0857b814 c8efa291fa31410fb999251c4c66d5af - - -] Exception during message handling
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 167, in wrapped
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 519, in wrapper
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3073, in update_access
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 280, in update_access_rules
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 319, in _update_access_rules
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 380, in _update_rules_through_share_driver
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     share_server=share_server
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 292, in update_access
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     access_key = self._allow_access(context, share, rule)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 241, in _allow_access
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     tenant_id=share['project_id'])
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 973, in authorize
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     key = self._authorize_volume(volume_path, auth_id, readonly)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1009, in _authorize_volume
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     key = self._authorize_ceph(volume_path, auth_id, readonly)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1086, in _authorize_ceph
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     'mon', cap['caps'].get('mon')]
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1275, in _rados_command
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server     raise rados.Error(outs)
2017-06-16 12:28:19.179 193387 ERROR oslo_messaging.rpc.server Error: Can't handle arrays of non-strings

------------------------------------

Then, when trying to deny access to the user, nothing happens, but share.log show the following error: 

------------------------------------

2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server [req-16ac61ab-1c5d-4cab-b685-e4eb030380e1 05e228f17d8b426b9071b1ee0857b814 c8efa291fa31410fb999251c4c66d5af - - -] Exception during message handling
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 167, in wrapped
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 519, in wrapper
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 3073, in update_access
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 280, in update_access_rules
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server=share_server)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 319, in _update_access_rules
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/access.py", line 380, in _update_rules_through_share_driver
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     share_server=share_server
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 296, in update_access
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     self._deny_access(context, share, rule)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/cephfs_native.py", line 253, in _deny_access
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     access['access_to'])
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 1133, in deauthorize
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server     self._recover_auth_meta(auth_id, auth_meta)
2017-06-16 12:30:56.036 193387 ERROR oslo_messaging.rpc.server TypeError: _recover_auth_meta() takes exactly 2 arguments (3 given)

------------------------------------


Version-Release number of selected component (if applicable):

puppet-manila-10.3.0-1.el7ost.noarch
openstack-manila-share-4.0.0-4.el7ost.noarch
python-manilaclient-1.14.0-1.el7ost.noarch
python-manila-4.0.0-4.el7ost.noarch
openstack-manila-4.0.0-4.el7ost.noarch
openstack-manila-ui-2.7.1-2.el7ost.noarch

Most relevant to the problem:
python-cephfs-10.2.5-37.el7cp.x86_64


Steps to Reproduce:
1. Create a user using the following command (note that alice could be any other user):

ceph --name=client.manila --keyring=/etc/ceph/ceph.client.manila.keyring auth get-or-create client.alice -o alice.keyring

2. Try to allow that user to have access to the share:

manila access-allow <share_name> cephx alice 

3. Run 'manila access-list <share>' and see that the share is in 'error' state

4. Try to run 'manila access-deny <share> <access-id>'


Actual results:

Share access in error state and error trace in /var/log/manila/share.log (in the node that runs manila share service)

Expected results:

Access removed successfully

---

Additional info:

The solution for this problem was addressed 8 months ago and is included in python-cephfs-10.2.7-27.el7cp.x86_64.rpm - we should make sure RHOS11 includes the package with the bugfix, or maybe backport the fix.

Comment 1 Tom Barron 2017-06-18 13:44:32 UTC

We have the same issue in OSP 10:

[root@compute-0 ~]# yum provides /usr/lib/python2.7/site-packages/ceph_volume_client.py
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
1:python-cephfs-10.2.5-37.el7cp.x86_64 : Python libraries for Ceph distributed file system
Repo        : @rhos-10.0-ceph-2.0-mon-signed
Matched from:
Filename    : /usr/lib/python2.7/site-packages/ceph_volume_client.py

Comment 2 Red Hat Bugzilla Rules Engine 2017-06-19 12:32:39 UTC

This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 5 Dustin Schoenbrun 2017-08-25 19:41:24 UTC

Using puddle version 2017-08-22.2 I was able to log into the undercloud node in a deploy and verify that the python-cephfs-10.2.7-27.el7cp.x86_64.rpm package (or greater) is available.

Comment 7 errata-xmlrpc 2017-09-06 17:06:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2653