I redeployed my test environment and confirmed the following items.
 - The issue is not reproduced when I use the default cluster name.
   I can set access for share without any error.

 - The issue is reproduced when I use a different cluster name,
   in this case central. I used the following template[1].
   The said error in manila-share is observed when I create a share[2],
   and it seems that ganesha failes to read ceph configuration properly[3].

[1]
parameter_defaults:
  CephClusterName: central
  GlanceBackend: rbd
  GlanceStoreDescription: 'central rbd glance store'
  GlanceBackendID: central
  CephClusterName: central

[2]
~~~
(overcloud) [stack@undercloud-0 ~]$ manila show share-01
+---------------------------------------+----------------------------------------------------------------------------+
| Property                              | Value                                                                      |
+---------------------------------------+----------------------------------------------------------------------------+
| id                                    | 3215bea1-b46d-4dee-a6ec-e93d3dae199a                                       |
| size                                  | 10                                                                         |
| availability_zone                     | nova                                                                       |
| created_at                            | 2020-12-03T14:11:39.000000                                                 |
| status                                | available                                                                  |
| name                                  | share-01                                                                   |
| description                           | None                                                                       |
| project_id                            | 1a94bfa09e3847a3a6f3a4402ef24a9a                                           |
| snapshot_id                           | None                                                                       |
| share_network_id                      | None                                                                       |
| share_proto                           | NFS                                                                        |
| metadata                              | {}                                                                         |
| share_type                            | 40633b9c-5210-4b06-8276-4052333bfdd2                                       |
| is_public                             | False                                                                      |
| snapshot_support                      | False                                                                      |
| task_state                            | None                                                                       |
| share_type_name                       | default                                                                    |
| access_rules_status                   | error                                                                      |
| replication_type                      | None                                                                       |
| has_replicas                          | False                                                                      |
| user_id                               | 7a5b51a64b7d468eb051d9fb31f08ad0                                           |
| create_share_from_snapshot_support    | False                                                                      |
| revert_to_snapshot_support            | False                                                                      |
| share_group_id                        | None                                                                       |
| source_share_group_snapshot_member_id | None                                                                       |
| mount_snapshot_support                | False                                                                      |
| share_server_id                       | None                                                                       |
| host                                  | hostgroup@cephfs#cephfs                                                    |
| export_locations                      |                                                                            |
|                                       | id = bc423940-16da-4fdd-a494-de7b59107cd8                                  |
|                                       | path = 172.17.5.123:/volumes/_nogroup/d7496598-3be7-4e70-af60-41c16cab5e9c |
|                                       | preferred = False                                                          |
|                                       | share_instance_id = d7496598-3be7-4e70-af60-41c16cab5e9c                   |
|                                       | is_admin_only = False                                                      |
+---------------------------------------+----------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ manila access-list share-01
+--------------------------------------+-------------+--------------+--------------+-------+------------+----------------------------+------------+
| id                                   | access_type | access_to    | access_level | state | access_key | created_at                 | updated_at |
+--------------------------------------+-------------+--------------+--------------+-------+------------+----------------------------+------------+
| 474cfec0-8b1a-4703-990d-c74e5ab99a33 | ip          | 172.17.5.160 | rw           | error | None       | 2020-12-03T14:11:57.000000 | None       |
+--------------------------------------+-------------+--------------+--------------+-------+------------+----------------------------+------------+
~~~

/var/log/containers/manila/manila-share.log
~~~
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 233, in _execute
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     return execute(*args, **kwargs)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 59, in __call__
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     return self.execute(*args, **exkwargs)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 101, in execute
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS string:EXPORT(Export_Id=1001)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Exit code: 1
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Stdout: ''
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidFileContent: 0 export entries in /etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS added because (export create, block validation) errors. Details:\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): 1 validation errors in block FSAL\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): Errors processing block (FSAL)\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): 1 validation errors in block EXPORT\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): Errors found in configuration block EXPORT\n\n'
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server 
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server 
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 474, in add_export
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     "string:EXPORT(Export_Id=%d)" % xid)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 414, in _dbus_send_ganesha
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     message='dbus call %s.%s' % (service, method), **kwargs)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 242, in _execute
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS string:EXPORT(Export_Id=1001)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Exit code: 1
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Stdout: ''
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidFileContent: 0 export entries in /etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS added because (export create, block validation) errors. Details:\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): 1 validation errors in block FSAL\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): Errors processing block (FSAL)\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): 1 validation errors in block EXPORT\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): Errors found in configuration block EXPORT\n\n'
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server 
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server 
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 187, in wrapped
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 3554, in update_access
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     share_server=share_server)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 283, in update_access_rules
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     share_server=share_server)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 322, in _update_access_rules
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     share_server)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     share_server=share_server
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     share_server=share_server)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/__init__.py", line 308, in update_access
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     self.ganesha.add_export(share['name'], confdict)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 491, in add_export
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS string:EXPORT(Export_Id=1001)
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Exit code: 1
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Stdout: ''
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidFileContent: 0 export entries in /etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS added because (export create, block validation) errors. Details:\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): 1 validation errors in block FSAL\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): Errors processing block (FSAL)\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): 1 validation errors in block EXPORT\nConfig File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): Errors found in configuration block EXPORT\n\n'
2020-12-03 14:01:10.402 44 ERROR oslo_messaging.rpc.server 
~~~

[3]
/var/log/ganesha/ganesha.log with debug enabled
~~~
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): do_block_init EXPORT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007548 name=Export_id type=CONFIG_UINT16
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007478 name=Path type=CONFIG_PATH
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007480 name=Pseudo type=CONFIG_PATH
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007498 name=MaxRead type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074a0 name=MaxWrite type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074a8 name=PrefRead type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074b0 name=PrefWrite type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074b8 name=PrefReaddir type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074d0 name=Filesystem_id type=CONFIG_FSID
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007488 name=Tag type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074c0 name=MaxOffsetWrite type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80074c8 name=MaxOffsetRead type=CONFIG_UINT64
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007540 name=UseCookieVerifier type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007540 name=DisableReaddirPlus type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007540 name=Trust_Readdir_Negative_Cache type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007540 name=Disable_ACL type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007540 name=Security_Label type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=Access_Type type=CONFIG_ENUM
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c CONFIG_ENUM Access_Type mask=000001e0 def=00000000 value=00000000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=Protocols type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c CONFIG_LIST Protocols mask=00700000 def=00300000 value=00300000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=Transports type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c CONFIG_LIST Transports mask=07000000 def=03000000 value=03300000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007520 name=Anonymous_uid type=CONFIG_ANON_ID
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007524 name=Anonymous_gid type=CONFIG_ANON_ID
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=SecType type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c CONFIG_LIST SecType mask=0001f000 def=00003000 value=03303000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=PrivilegedPort type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=Manage_Gids type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=Squash type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c CONFIG_LIST Squash mask=00000007 def=00000002 value=03303002
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=NFS_Commit type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c name=Delegations type=CONFIG_ENUM
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800752c CONFIG_ENUM Delegations mask=30000000 def=00000000 value=03303002
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007528 name=Attr_Expiration_Time type=CONFIG_INT32
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007450 name=Client type=CONFIG_BLOCK
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007470 name=FSAL type=CONFIG_BLOCK
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): do_block_load EXPORT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007548 name=Export_id type=CONFIG_UINT16
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007478 name=Path type=CONFIG_PATH
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007480 name=Pseudo type=CONFIG_PATH
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007488 name=Tag type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a800752c name=SecType type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a800752c CONFIG_LIST SecType mask=0001f000 flags=00002000 value=03302002
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a800752c name=Squash type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a800752c CONFIG_LIST Squash mask=00000007 flags=00000000 value=03302000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007450 name=Client type=CONFIG_BLOCK
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:12): do_block_init Client
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=Access_Type type=CONFIG_ENUM
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c CONFIG_ENUM Access_Type mask=000001e0 def=00000000 value=00000000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=Protocols type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c CONFIG_LIST Protocols mask=00700000 def=00300000 value=00300000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=Transports type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c CONFIG_LIST Transports mask=07000000 def=03000000 value=03300000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007600 name=Anonymous_uid type=CONFIG_ANON_ID
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007604 name=Anonymous_gid type=CONFIG_ANON_ID
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=SecType type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c CONFIG_LIST SecType mask=0001f000 def=00003000 value=03303000
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=PrivilegedPort type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=Manage_Gids type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=Squash type=CONFIG_LIST
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c CONFIG_LIST Squash mask=00000007 def=00000002 value=03303002
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=NFS_Commit type=CONFIG_BOOLBIT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c name=Delegations type=CONFIG_ENUM
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a800760c CONFIG_ENUM Delegations mask=30000000 def=00000000 value=03303002
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a80075e0 name=Clients type=CONFIG_PROC
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:12): do_block_load Client
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a800760c name=Access_Type type=CONFIG_ENUM
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a800760c CONFIG_ENUM Access_Type mask=000001e0 flags=000001e0 value=033031e2
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a80075e0 name=Clients type=CONFIG_PROC
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:12): commit Client
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007470 name=FSAL type=CONFIG_BLOCK
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] fsal_init :CONFIG :F_DBG :Allocating args 0x7ff5a8007470/0x7ff5a8007690
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): do_block_init FSAL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8007690 name=Name type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): do_block_load FSAL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8007690 name=Name type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): commit FSAL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5e45d76a8 name=ceph_conf type=CONFIG_PATH
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5e45d74e0 name=umask type=CONFIG_UINT32
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): do_block_init FSAL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8004b50 name=name type=CONFIG_NULL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8004ca0 name=user_id type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8004cb8 name=filesystem type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8004ca8 name=secret_access_key type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_init :CONFIG :F_DBG :0x7ff5a8004cb0 name=sec_label_xattr type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): do_block_load FSAL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8004b50 name=name type=CONFIG_NULL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8004ca0 name=user_id type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] do_block_load :CONFIG :F_DBG :0x7ff5a8004ca8 name=secret_access_key type=CONFIG_STRING
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): commit FSAL
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] create_export :FSAL :CRIT :Unable to read Ceph configuration for /volumes/_nogroup/e3fb8b58-c367-464e-8639-7a9dd143f11c.
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/volumes/_nogroup/e3fb8b58-c367-464e-8639-7a9dd143f11c) to (/volumes/_nogroup/e3fb8b58-c367-464e-8639-7a9dd143f11c)
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :Releasing block 0x7ff5a8007470/0x7ff5a8007690
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :------ At (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): commit EXPORT
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] proc_block :CONFIG :F_DBG :Releasing block (nil)/0x7ff5a80073a8
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] gsh_export_addexport :EXPORT :CRIT :0 export entries in /etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS added because (export create, block validation) errors
03/12/2020 14:01:10 : epoch 5fc8ee9e : controller-0 : ganesha.nfsd-58[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (AddExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidFileContent), message = (0 export entries in /etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS added because (export create, block validation) errors. Details:
Config File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): 1 validation errors in block FSAL
Config File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:4): Errors processing block (FSAL)
Config File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): 1 validation errors in block EXPORT
Config File (/etc/ganesha/export.d/share-e3fb8b58-c367-464e-8639-7a9dd143f11c.conf.yJEJnS:1): Errors found in configuration block EXPORT
~~`

I tried creating an access rule after adding the following ceph_conf parameter in /etc/ganesha/ganesha.conf
but I still observe the same error.

~~~
NFS_Core_Param
{
       Bind_Addr=172.17.5.123;
}

EXPORT_DEFAULTS {
        Attr_Expiration_Time = 0;
}

CACHEINODE {
        Dir_Chunk = 0;

        NParts = 1;
        Cache_Size = 1;
}

RADOS_URLS {
   ceph_conf = '/etc/ceph/central.conf';
   userid = "manila";
}
%url rados://manila_data/ganesha-export-index

NFSv4 {
        RecoveryBackend = 'rados_kv';
}
RADOS_KV {
        ceph_conf = '/etc/ceph/central.conf';
        userid = "manila";
        pool = "manila_data";
}

CEPH {
        ceph_conf = "/etc/ceph/central.conf"; <====(*)
}

LOG {
        Facility {
                name = FILE;
                destination = "/var/log/ganesha/ganesha.log";
                enable = active;
        }

        Components {
                CONFIG=FULL_DEBUG;
        }
}
~~~

(In reply to Takashi Kajinami from comment #6)
> I tried creating an access rule after adding the following ceph_conf
> parameter in /etc/ganesha/ganesha.conf
> but I still observe the same error.
> 
> ~~~
> NFS_Core_Param
> {
>        Bind_Addr=172.17.5.123;
> }
> 
> EXPORT_DEFAULTS {
>         Attr_Expiration_Time = 0;
> }
> 
> CACHEINODE {
>         Dir_Chunk = 0;
> 
>         NParts = 1;
>         Cache_Size = 1;
> }
> 
> RADOS_URLS {
>    ceph_conf = '/etc/ceph/central.conf';
>    userid = "manila";
> }
> %url rados://manila_data/ganesha-export-index
> 
> NFSv4 {
>         RecoveryBackend = 'rados_kv';
> }
> RADOS_KV {
>         ceph_conf = '/etc/ceph/central.conf';
>         userid = "manila";
>         pool = "manila_data";
> }
> 
> CEPH {
>         ceph_conf = "/etc/ceph/central.conf"; <====(*)
> }
> 
> LOG {
>         Facility {
>                 name = FILE;
>                 destination = "/var/log/ganesha/ganesha.log";
>                 enable = active;
>         }
> 
>         Components {
>                 CONFIG=FULL_DEBUG;
>         }
> }
> ~~~

Thank you for the great debugging!

We will use this information to set up an environment where we can reproduce the issue.
Manila dynamically generates the EXPORTS block and we need to investigate whether we
need to do this a bit differently when the cluster name isn't 'ceph', whether we need
to adjust the static part of ganesha.conf, or whether there is an underlying issue in
ganesha itself.  Your debugging effort really helps us frame the investigation.

https://bugzilla.redhat.com/show_bug.cgi?id=1901143 is probably a duplicate.  Good news if so is that we can reproduce locally.  See: https://bugzilla.redhat.com/show_bug.cgi?id=1901143#c7

Created attachment 1738208 [details]
ganesha.conf and ganesha.log with CEPH config block

That syntax looks correct, mostly -- not sure about the newline between CEPH and the open brace, but I think that's allowed.

I'll note that this is in the logfile in the tarball:

    09/12/2020 22:00:56 : epoch 5fd14918 : ubuntu-1 : ganesha.nfsd-328176[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!

If there are no FSAL_CEPH exports, then the CEPH block probably won't be loaded (because the driver isn't used). Is there a FSAL_CEPH export in
rados://cephfs_data/ganesha-export-index ?

(In reply to Jeff Layton from comment #12)
> That syntax looks correct, mostly -- not sure about the newline between CEPH
> and the open brace, but I think that's allowed.
> 
> I'll note that this is in the logfile in the tarball:
> 
>     09/12/2020 22:00:56 : epoch 5fd14918 : ubuntu-1 :
> ganesha.nfsd-328176[main] main :NFS STARTUP :WARN :No export entries found
> in configuration file !!!
> 
> If there are no FSAL_CEPH exports, then the CEPH block probably won't be
> loaded (because the driver isn't used). Is there a FSAL_CEPH export in
> rados://cephfs_data/ganesha-export-index ?

I adjusted the syntax to remove the newline between CEPH and the open brace with the same results.

The 'No export entries found ...!' message in the ganesha log at startup is normal for Manila.  The manila service
and nfs-ganesha first start up without users having created or allowed access to shares.  Later, a user 'allow access'
command results in a dbus message to ganesha with an EXPORT config block.  These are not written to /etc/ganesha/ganesha.conf;
the file itself it never has EXPORT blocks in its configuration but as you know they are peristed in RADOS.

At startup if I do 'sudo rados -c /etc/ceph/CephXXX.conf -p cephfs_data ls' I see one object, ganesha-export-index.
When I do ''sudo rados -c /etc/ceph/CephXXX.conf -p cephfs_data get ganesha-export-index output' the file *output*
is created but it is empty.

If I successfully allow access to a share, then ganesha-export-index contents are like this:

  url rados://cephfs_data/ganesha-export-share-f1b577d5-00d0-429e-95a9-458c678abf81

and there is a corresponding object in the same cephfs_data pool which contains the actual EXPORT config:

EXPORT {
    Export_Id = 1001;
    Path = "/volumes/_nogroup/f1b577d5-00d0-429e-95a9-458c678abf81";
    FSAL {
        Name = "Ceph";
        User_Id = "ganesha-f1b577d5-00d0-429e-95a9-458c678abf81";
        Secret_Access_Key = "AQA9eNJf2Q/9LRAAI0MaaTjCfPBU75bsOI4xEQ==";
    }
    Pseudo = "/volumes/_nogroup/f1b577d5-00d0-429e-95a9-458c678abf81";
    SecType = "sys";
    Tag = "share-f1b577d5-00d0-429e-95a9-458c678abf81";
    CLIENT {
        Access_Type = "rw";
        Clients = 0.0.0.0;
    }

    Squash = "None";
}

So yeah, it may be that because the EXPORTS are "added" by dbus that the CEPH block in ganesha.conf isn't being parsed and loaded.

Now if I 'sudo systemctl restart nfs-ganesha' and examine /var/log/ganesha.log I indeed see the rados:// based config being merged in and 'ceph_conf'
being parsed and loaded in the context of a 'Ceph' block, not just in the context of the RADOS_URLs block.  The "pump has been primed."

Indeed, if I allow access for that share from a new IP or CIDR at this point that action succeeds!

So is there some way we can trick ganesha into parsing and loading the top level CEPH block before there are any exports processed?

That's a normal startup scenario for Ganesha with Manila.

Is there a reason this is private? We should open this up.

I would avoid dbus for this.

Once you persist these things in RADOS, you should send the daemon a SIGHUP.

Or, better yet, consider setting up the watch_url setting, and send a RADOS notify to all watchers. See:

    https://github.com/nfs-ganesha/nfs-ganesha/blob/f3cb6b9ac21a4791d09cf77c5fa35a896e2b803d/src/config_samples/ceph.conf#L208

It seems like this behavior may be due to use of dbus to signal ganesha that it should pick up EXPORT config blocks persisted in RADOS and
Jef Layton has suggested that we may be able instead just to send the ganesha daemon a SIGHUP.  I guess that would fit with my experiment
doing a 'systemctl restart' on the service, and I like the simplification, but Jeff I don't see how that gets us past the initial "prime the
pump" issue.  As I mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1903005#c13, when Ganesha and Manila start up there is just an
empty ganesha-export-index object and processing it does not seem to be sufficient to get the top level CEPH block parsed.  I can *imagine*
manila making some dummy share and dummy export at startup (that are never shown to the user) and then sending a SIGHUP to ganesha but that
seems a bit of a kludgy, manila-specific workaround that wouldn't solve the same problem for a different ganesha consumer than manila.
I wonder if this issue is structurally similar to this older issue with ceph_conf etc. in the RADOS_URLS config block:
https://bugzilla.redhat.com/show_bug.cgi?id=1486112

I'm not sure why you'd need to "prime the pump". When you send the daemon a SIGHUP, the base configuration file is reread. It should pick up the CEPH block when it rereads the configuration. This is different from a systemctl restart which is going to kill the daemon and restart it (and kick off a grace period).

The watch_url setting is what you really want to use. That will make ganesha watch that object. After you update the configuration, send the watch object a RADOS notify. The daemons treat that like a SIGHUP and will reread their whole config. That way, you notify all the daemons watching that URL with one shot.

Something like this in the RADOS_URLS config block:

    rados://manila_data/ganesha-export-index

I'm not clear on how manila updates objects, but it should be configured to issue a RADOS notify to the above object once the changes are done.

Sorry, something like:

    watch_url = "rados://manila_data/ganesha-export-index";

@Takashi Kajinami - Until we get manila and ganesha sorted, I think your customer can use a simple workaround: on each Controller node, run 'sudo ln /etc/ceph/central.conf /etc/ceph/ceph.conf'  That way the parts of ganesha like the rados url lookups that are expecting to see the config in the configured /etc/ceph/central.conf will find it there and the Ceph FSAL exports will find the same configuration contents under the unconfigured but default ceph.conf path.  This works for me in an upstream development environment but please confirm that it will work for the customer.  It *should* since the entire /etc/ceph directory from the host is mounted in the containers where ganesha (ceph-nfs) and manila-share services run.

@Tom,

Thank you for your inputs. I'll discuss the solution with the customer.
I thought that another solution would be stick to the default name "ceph",
but I feel your solution is better since the name "central" is what we
usually expect in DCN deployment.

FYI.

I tested Manila functionality after implementing the following workaround,
and confirmed that now manila can create an export in ganesha.

1. Create a symbolic link in all controller nodes so that /etc/ceph/central.conf
   can be accessed by /etc/ceph/ceph.conf
 $ sudo ln -s /etc/ceph/central.conf /etc/ceph/ceph.conf

2. Restart openstack-manila-share resource
 $ sudo pcs resource restart openstack-manila-share

I'll provide the above steps to the customer.
# You mentioned hard link but it seems symbolic link is enough in this case.

(In reply to Takashi Kajinami from comment #21)
> FYI.
> 
> I tested Manila functionality after implementing the following workaround,
> and confirmed that now manila can create an export in ganesha.
> 
> 1. Create a symbolic link in all controller nodes so that
> /etc/ceph/central.conf
>    can be accessed by /etc/ceph/ceph.conf
>  $ sudo ln -s /etc/ceph/central.conf /etc/ceph/ceph.conf
> 
> 2. Restart openstack-manila-share resource
>  $ sudo pcs resource restart openstack-manila-share
> 
> I'll provide the above steps to the customer.
> # You mentioned hard link but it seems symbolic link is enough in this case.

Thanks for confirming.  I first wrote the workaround as a symboliic link, then changed it to a hard link just to emphasize that the two files are exactly the same under a different path.  They just need to be in the same directory /etc/ceph (on Red Hat with containerized services) but a symbolic link or hard link or even a copy should work.

*** Bug 1901143 has been marked as a duplicate of this bug. ***

*** Bug 1970427 has been marked as a duplicate of this bug. ***

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days