1639823 – Using CephNFS with NFS (Ganesha) is not possible to create the share due to incorrect permission

Bug 1639823 - Using CephNFS with NFS (Ganesha) is not possible to create the share due to incorrect permission

Summary: Using CephNFS with NFS (Ganesha) is not possible to create the share due to i...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-manila
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	low
Target Milestone:	---
Target Release:	---
Assignee:	Tom Barron
QA Contact:	Dustin Schoenbrun
Docs Contact:	mmurray
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-10-16 16:56 UTC by Alberto Gonzalez
Modified:	2018-10-25 15:51 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-10-25 15:51:26 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Alberto Gonzalez 2018-10-16 16:56:56 UTC

Description of problem:
Creating a share throws following error:

2018-10-16 11:34:07.414 22 ERROR manila.share.manager [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80 94b36491febf41ee90d1db714da46747 - - -] Share instance 6324878f-4d08-4027-aaa6-b3a0a2369628 failed on creation.: Error: (13, "error in mkdir 'volumes': error code 13")
2018-10-16 11:34:07.414 22 WARNING manila.share.manager [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80 94b36491febf41ee90d1db714da46747 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: Error: (13, "error in mkdir 'volumes': error code 13")
2018-10-16 11:34:07.446 22 INFO manila.message.api [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80 94b36491febf41ee90d1db714da46747 - - -] Creating message record for request_id = req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80 94b36491febf41ee90d1db714da46747 - - -] Exception during message handling: Error: (13, "error in mkdir 'volumes': error code 13")
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 187, in wrapped
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/utils.py", line 571, in wrapper
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1762, in create_share_instance
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     exception=e)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1725, in create_share_instance
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     context, share_instance, share_server=share_server)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py", line 234, in create_share
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     cephfs_share_path(share), size=size, data_isolated=data_isolated)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 616, in create_volume
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     self._mkdir_p(path)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 599, in _mkdir_p
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     self.fs.mkdir(subpath, 0o755)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File "cephfs.pyx", line 628, in cephfs.LibCephFS.mkdir (/builddir/build/BUILD/ceph-12.2.4/build/src/pybind/cephfs/pyrex/cephfs.c:7490)
2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server Error: (13, "error in mkdir 'volumes': error code 13")

Version-Release number of selected component (if applicable):
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp13/openstack-manila-share/images/13.0-54",


How reproducible:
Create a share using manila command

Steps to Reproduce:
1. Run manila create --name testnfs1 nfs 1
2. Check share is on error status

Actual results:
Share is on error status

Expected results:
Share is created correctly

Additional info:
There is a workaround for this issue

[root@overcloud-controller-0 ~]# mount -t ceph 172.20.0.33:6789:/ /mnt -o name=manila,secret=<your manila client key>

[root@overcloud-controller-0 ~]# chown 42429:42429 /mnt/
[root@overcloud-controller-0 ~]# umount /mnt/

172.20.0.33 is the VIP address for ceph_nfs

Comment 1 Tom Barron 2018-10-17 01:30:45 UTC

(In reply to Alberto Gonzalez from comment #0)
> Description of problem:
> Creating a share throws following error:
> 
> 2018-10-16 11:34:07.414 22 ERROR manila.share.manager
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Share instance
> 6324878f-4d08-4027-aaa6-b3a0a2369628 failed on creation.: Error: (13, "error
> in mkdir 'volumes': error code 13")
> 2018-10-16 11:34:07.414 22 WARNING manila.share.manager
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Share instance information in
> exception can not be written to db because it contains {} and it is not a
> dictionary.: Error: (13, "error in mkdir 'volumes': error code 13")
> 2018-10-16 11:34:07.446 22 INFO manila.message.api
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Creating message record for
> request_id = req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Exception during message handling:
> Error: (13, "error in mkdir 'volumes': error code 13")
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server Traceback (most
> recent call last):
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163,
> in _process_incoming
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     res =
> self.dispatcher.dispatch(message)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 220, in dispatch
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 190, in _do_dispatch
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     result =
> func(ctxt, **new_args)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 187, in
> wrapped
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return
> f(self, *args, **kwargs)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/utils.py", line 571, in wrapper
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return
> func(self, *args, **kwargs)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1762, in
> create_share_instance
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     exception=e)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> self.force_reraise()
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> six.reraise(self.type_, self.value, self.tb)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1725, in
> create_share_instance
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     context,
> share_instance, share_server=share_server)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py",
> line 234, in create_share
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> cephfs_share_path(share), size=size, data_isolated=data_isolated)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 616, in
> create_volume
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> self._mkdir_p(path)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 599, in
> _mkdir_p
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> self.fs.mkdir(subpath, 0o755)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "cephfs.pyx", line 628, in cephfs.LibCephFS.mkdir
> (/builddir/build/BUILD/ceph-12.2.4/build/src/pybind/cephfs/pyrex/cephfs.c:
> 7490)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server Error: (13,
> "error in mkdir 'volumes': error code 13")
> 
> Version-Release number of selected component (if applicable):
>                 "url":
> "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp13/
> openstack-manila-share/images/13.0-54",
> 
> 
> How reproducible:
> Create a share using manila command
> 
> Steps to Reproduce:
> 1. Run manila create --name testnfs1 nfs 1
> 2. Check share is on error status
> 
> Actual results:
> Share is on error status
> 
> Expected results:
> Share is created correctly
> 
> Additional info:
> There is a workaround for this issue
> 
> [root@overcloud-controller-0 ~]# mount -t ceph 172.20.0.33:6789:/ /mnt -o
> name=manila,secret=<your manila client key>
> 
> [root@overcloud-controller-0 ~]# chown 42429:42429 /mnt/
> [root@overcloud-controller-0 ~]# umount /mnt/
> 
> 172.20.0.33 is the VIP address for ceph_nfs

Alberto, is there any chance I can get access to this system?  This looks like a packaging mismatch, we have had some of these since OSP13 shipped, and the quickest thing may be for me to poke around a bit on your system (I won't change anything).

Comment 5 Tom Barron 2018-10-17 22:23:10 UTC

Alberto did the solution for https://bugzilla.redhat.com/show_bug.cgi?id=1639822 also work here?  Looks like shares are successfully being created now ...

Comment 6 Alberto Gonzalez 2018-10-25 15:51:26 UTC

Solved with  https://bugzilla.redhat.com/show_bug.cgi?id=1639822 
Removing -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/manila.yaml from the installation solves this issue. This template should be only used during the image prepare for the containers and not for the installation.

Note You need to log in before you can comment on or make changes to this bug.