Bug 1639823

(In reply to Alberto Gonzalez from comment #0)
> Description of problem:
> Creating a share throws following error:
> 
> 2018-10-16 11:34:07.414 22 ERROR manila.share.manager
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Share instance
> 6324878f-4d08-4027-aaa6-b3a0a2369628 failed on creation.: Error: (13, "error
> in mkdir 'volumes': error code 13")
> 2018-10-16 11:34:07.414 22 WARNING manila.share.manager
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Share instance information in
> exception can not be written to db because it contains {} and it is not a
> dictionary.: Error: (13, "error in mkdir 'volumes': error code 13")
> 2018-10-16 11:34:07.446 22 INFO manila.message.api
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Creating message record for
> request_id = req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server
> [req-f3c45977-7b73-41aa-8fcc-08d11c44ffb1 a7ebf28c9453482caaad1044cc619e80
> 94b36491febf41ee90d1db714da46747 - - -] Exception during message handling:
> Error: (13, "error in mkdir 'volumes': error code 13")
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server Traceback (most
> recent call last):
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163,
> in _process_incoming
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     res =
> self.dispatcher.dispatch(message)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 220, in dispatch
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 190, in _do_dispatch
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     result =
> func(ctxt, **new_args)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 187, in
> wrapped
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return
> f(self, *args, **kwargs)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/utils.py", line 571, in wrapper
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     return
> func(self, *args, **kwargs)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1762, in
> create_share_instance
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     exception=e)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> self.force_reraise()
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> six.reraise(self.type_, self.value, self.tb)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 1725, in
> create_share_instance
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server     context,
> share_instance, share_server=share_server)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py",
> line 234, in create_share
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> cephfs_share_path(share), size=size, data_isolated=data_isolated)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 616, in
> create_volume
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> self._mkdir_p(path)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python2.7/site-packages/ceph_volume_client.py", line 599, in
> _mkdir_p
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server    
> self.fs.mkdir(subpath, 0o755)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server   File
> "cephfs.pyx", line 628, in cephfs.LibCephFS.mkdir
> (/builddir/build/BUILD/ceph-12.2.4/build/src/pybind/cephfs/pyrex/cephfs.c:
> 7490)
> 2018-10-16 11:34:07.461 22 ERROR oslo_messaging.rpc.server Error: (13,
> "error in mkdir 'volumes': error code 13")
> 
> Version-Release number of selected component (if applicable):
>                 "url":
> "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp13/
> openstack-manila-share/images/13.0-54",
> 
> 
> How reproducible:
> Create a share using manila command
> 
> Steps to Reproduce:
> 1. Run manila create --name testnfs1 nfs 1
> 2. Check share is on error status
> 
> Actual results:
> Share is on error status
> 
> Expected results:
> Share is created correctly
> 
> Additional info:
> There is a workaround for this issue
> 
> [root@overcloud-controller-0 ~]# mount -t ceph 172.20.0.33:6789:/ /mnt -o
> name=manila,secret=<your manila client key>
> 
> [root@overcloud-controller-0 ~]# chown 42429:42429 /mnt/
> [root@overcloud-controller-0 ~]# umount /mnt/
> 
> 172.20.0.33 is the VIP address for ceph_nfs

Alberto, is there any chance I can get access to this system?  This looks like a packaging mismatch, we have had some of these since OSP13 shipped, and the quickest thing may be for me to poke around a bit on your system (I won't change anything).

Alberto did the solution for https://bugzilla.redhat.com/show_bug.cgi?id=1639822 also work here?  Looks like shares are successfully being created now ...

Solved with  https://bugzilla.redhat.com/show_bug.cgi?id=1639822 
Removing -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/manila.yaml from the installation solves this issue. This template should be only used during the image prepare for the containers and not for the installation.