Bug 1993210

Summary: [Test Only] Couldn't create Manila shares - ceph-nfs-pacemaker is dumped
Product: Red Hat OpenStack Reporter: Victoria Martinez de la Cruz <vimartin>
Component: openstack-manilaAssignee: Victoria Martinez de la Cruz <vimartin>
Status: CLOSED CURRENTRELEASE QA Contact: vhariria
Severity: high Docs Contact: mmurray
Priority: high    
Version: 17.0 (Wallaby)CC: dsavinea, gfidente, gouthamr, jinjli, jlayton, mbenjamin, mmurray, pasik, pdhange, pdonnell, pgrist, rraja, vhariria, vimartin, yocha
Target Milestone: ---Keywords: Scale, TestOnly, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1987235 Environment:
Last Closed: 2022-06-02 09:43:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1987235, 1992473    
Bug Blocks:    

Description Victoria Martinez de la Cruz 2021-08-12 14:40:58 UTC 
+++ This bug was initially created as a clone of Bug #1987235 +++

Description of problem:
 - ceph-nfs are get hung status when try to create or set more share rules. 
 - it recovered when run "pcs resource cleanup"

Version-Release number of selected component :

- Red Hat OpenStack Platform release 16.1.6 GA (Train)
- RHCS 4.2

- using ganesha for manila backend.

c3a28484bab4  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              20 hours ago  Up 20 hours ago         ceph-nfs-pacemaker
b3e14e7e9f90  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-mds-dint-sdc-ctrl-03
2d321e9ff8e6  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-mon-dint-sdc-ctrl-03
4b09170ff142  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-mgr-dint-sdc-ctrl-03
3e315954bb08  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-crash-dint-sdc-ctrl-03

podman_inspect_172.16.222.212_5000.rhceph.rhceph-4-rhel8_latest:                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/4-54",
podman_inspect_172.16.222.212_5000.rhceph.rhceph-4-rhel8_latest:            "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/4-54",


Number of share volume : 485
volumes that using share rule : 270


How reproducible:

- it is occurring when they trying to create more share rules.


Actual results:

- ganesha.nfsd was crashed when set share rules. 

-From the messages log.

==>> Jul 27 14:37:40 dint-sdc-ctrl-03 systemd-coredump[342454]: Process 195537 (ganesha.nfsd) of user 0 dumped core.#012#012Stack trace of thread 5184:#012#0  0x00007f20efc1284f n/a (n/a)
...
Jul 27 14:37:40 dint-sdc-ctrl-03 podman[343693]: 2021-07-27 14:37:40.734824275 +0900 KST m=+0.044091204 container died 92fa9d5b889b5f9d95e7d0889c814127c0be7bb4ece4385d42a62606704e32e8 (image=172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest, name=ceph-nfs-pacemaker)
Jul 27 14:37:40 dint-sdc-ctrl-03 podman[343693]: 2021-07-27 14:37:40.790747921 +0900 KST m=+0.100014853 container remove 92fa9d5b889b5f9d95e7d0889c814127c0be7bb4ece4385d42a62606704e32e8 (image=172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest, name=ceph-nfs-pacemaker)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Result of monitor operation for ceph-nfs on dint-sdc-ctrl-03: 7 (not running)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Transition 5407 action 19 (ceph-nfs_monitor_60000 on dint-sdc-ctrl-03): expected 'ok' but got 'not running'
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-attrd[107321]: notice: Setting fail-count-ceph-nfs#monitor_60000[dint-sdc-ctrl-03]: 5 -> 6
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-attrd[107321]: notice: Setting last-failure-ceph-nfs#monitor_60000[dint-sdc-ctrl-03]: 1627276358 -> 1627364269
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: warning: Unexpected result (not running) was recorded for monitor of ceph-nfs on dint-sdc-ctrl-03 at Jul 27 14:37:49 2021
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: notice:  * Recover    ceph-nfs                            (                     dint-sdc-ctrl-03 )
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: warning: Unexpected result (not running) was recorded for monitor of ceph-nfs on dint-sdc-ctrl-03 at Jul 27 14:37:49 2021
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: notice:  * Recover    ceph-nfs                            (                     dint-sdc-ctrl-03 )
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Initiating stop operation ceph-nfs_stop_0 locally on dint-sdc-ctrl-03
Jul 27 14:37:51 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Result of stop operation for ceph-nfs on dint-sdc-ctrl-03: 0 (ok)
Jul 27 14:37:51 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Initiating start operation ceph-nfs_start_0 locally on dint-sdc-ctrl-03

- From the pacemaker

Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_process_request)     info: Forwarding cib_modify operation for section status to all (origin=local/crmd/6387)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: Diff: --- 0.193.87 2
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: Diff: +++ 0.193.88 (null)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: +  /cib:  @num_updates=88
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: +  /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ceph-nfs']/lrm_rsc_op[@id='ceph-nfs_last_failure_0']:  @transition-key=19:5407:0:97933452-33ab-49ab-b724-071a12dc7ba0, @transition-magic=0:7;19:5407:0:97933452-33ab-49ab-b724-071a12dc7ba0, @call-id=347, @last-rc-change=1627364269
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_process_request)     info: Completed cib_modify operation for section status: OK (rc=0, origin=dint-sdc-ctrl-03/crmd/6387, version=0.193.88)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld  [107324] (abort_transition_graph)  info: Transition 5504 aborted by operation ceph-nfs_monitor_60000 'modify' on dint-sdc-ctrl-03: Change in recurring result | magic=0:7;19:5407:0:97933452-33ab-49ab-b724-071a12dc7ba0 cib=0.193.88 source=process_graph_event:406 complete=true
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld  [107324] (update_failcount)        info: Updating failcount for ceph-nfs on dint-sdc-ctrl-03 after failed monitor: rc=7 (update=value++, time=1627364269)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld  [107324] (process_graph_event)     notice: Transition 5407 action 19 (ceph-nfs_monitor_60000 on dint-sdc-ctrl-03): expected 'ok' but got 'not running' | target-rc=0 rc=7 call-id=347 event='arrived after initial scheduling'


- From the manila-share.log.1

2021-07-27 14:37:40.626 42 ERROR manila.share.drivers.ganesha.manager [req-566e8125-cfe4-4c37-8f0d-5a388faeaff2 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.AddExport.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server [req-566e8125-cfe4-4c37-8f0d-5a388faeaff2 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 233, in _execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return execute(*args, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 59, in __call__
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return self.execute(*args, **exkwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 101, in execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 474, in add_export
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     "string:EXPORT(Export_Id=%d)" % xid)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 414, in _dbus_send_ganesha
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     message='dbus call %s.%s' % (service, method), **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 242, in _execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 187, in wrapped
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 3554, in update_access
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 283, in update_access_rules
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 322, in _update_access_rules
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/__init__.py", line 308, in update_access
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     self.ganesha.add_export(share['name'], confdict)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 491, in add_export
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:38:07.805 42 INFO manila.share.manager [req-87fe4d74-3c50-43bc-9d94-a39aaeb58bbc - - - - -] Updating share status
2021-07-27 14:38:31.990 42 ERROR manila.share.drivers.ganesha.manager [req-907e91a1-85e9-4a9a-8134-4cfeeee45890 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.AddExport.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server [req-907e91a1-85e9-4a9a-8134-4cfeeee45890 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-eecb4779-ea86-418b-beb7-5a4651873be2.conf.J1j2w9 string:EXPORT(Export_Id=1539)
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 233, in _execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     return execute(*args, **kwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 59, in __call__
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     return self.execute(*args, **exkwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 101, in execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-eecb4779-ea86-418b-beb7-5a4651873be2.conf.J1j2w9 string:EXPORT(Export_Id=1539)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 474, in add_export
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     "string:EXPORT(Export_Id=%d)" % xid)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 414, in _dbus_send_ganesha
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     message='dbus call %s.%s' % (service, method), **kwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 242, in _execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-eecb4779-ea86-418b-beb7-5a4651873be2.conf.J1j2w9 string:EXPORT(Export_Id=1539)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'


Expected results:
- Add normaly more share rules.

Additional info:

- Asked ceph cluster status when hit issue, CU said it wasn't effect to ceph cluster status.

- You can find related data in below

$ ssh <your-id>@supportshell.cee.redhat.com
$ yank 02997902
$ cd 02997902
[yocha@supportshell 02997902]$ ll
total 896932
-rw-rwxrw-+  1 yank  yank     59794 Jul 28 11:44 0010-manila-error.txt
....
-rw-rw-rw-+  1 yank  yank     62488 Jul 29 07:57 0070-systemd-coredump                                 <<== core file. 
drwxrwxrwx+  2 yank  yank      4096 Jul 29 08:34 0080-core.ganesha.tar.gz                              <<== core file.
drwxrwxrwx+ 13 yocha yocha     4096 Jul 28 02:50 sosreport-dint-sdc-ctrl-03-2021-07-28-anuqybu         <<== sosreport from controller node.
drwxrwxrwx+  3 yank  yank        67 Jul 29 00:24 sosreport-dint-sdc-ctrl-03-2021-07-28-anuqybu.tar
....

Let me know if you need addtional info.

Thank you,
Regards,
YoungCheol