1993210 – [Test Only] Couldn't create Manila shares - ceph-nfs-pacemaker is dumped

Bug 1993210 - [Test Only] Couldn't create Manila shares - ceph-nfs-pacemaker is dumped

Summary: [Test Only] Couldn't create Manila shares - ceph-nfs-pacemaker is dumped

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-manila
Sub Component:
Version:	17.0 (Wallaby)
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Victoria Martinez de la Cruz
QA Contact:	vhariria
Docs Contact:	mmurray
URL:
Whiteboard:
Depends On:	1987235 1992473
Blocks:
TreeView+	depends on / blocked

Reported:	2021-08-12 14:40 UTC by Victoria Martinez de la Cruz
Modified:	2022-06-02 09:43 UTC (History)
CC List:	15 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1987235
Environment:
Last Closed:	2022-06-02 09:43:06 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-7069	0	None	None	None	2021-11-15 12:57:23 UTC

Description Victoria Martinez de la Cruz 2021-08-12 14:40:58 UTC

+++ This bug was initially created as a clone of Bug #1987235 +++

Description of problem:
 - ceph-nfs are get hung status when try to create or set more share rules. 
 - it recovered when run "pcs resource cleanup"

Version-Release number of selected component :

- Red Hat OpenStack Platform release 16.1.6 GA (Train)
- RHCS 4.2

- using ganesha for manila backend.

c3a28484bab4  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              20 hours ago  Up 20 hours ago         ceph-nfs-pacemaker
b3e14e7e9f90  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-mds-dint-sdc-ctrl-03
2d321e9ff8e6  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-mon-dint-sdc-ctrl-03
4b09170ff142  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-mgr-dint-sdc-ctrl-03
3e315954bb08  172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest                                              6 weeks ago   Up 6 weeks ago          ceph-crash-dint-sdc-ctrl-03

podman_inspect_172.16.222.212_5000.rhceph.rhceph-4-rhel8_latest:                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/4-54",
podman_inspect_172.16.222.212_5000.rhceph.rhceph-4-rhel8_latest:            "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/4-54",


Number of share volume : 485
volumes that using share rule : 270


How reproducible:

- it is occurring when they trying to create more share rules.


Actual results:

- ganesha.nfsd was crashed when set share rules. 

-From the messages log.

==>> Jul 27 14:37:40 dint-sdc-ctrl-03 systemd-coredump[342454]: Process 195537 (ganesha.nfsd) of user 0 dumped core.#012#012Stack trace of thread 5184:#012#0  0x00007f20efc1284f n/a (n/a)
...
Jul 27 14:37:40 dint-sdc-ctrl-03 podman[343693]: 2021-07-27 14:37:40.734824275 +0900 KST m=+0.044091204 container died 92fa9d5b889b5f9d95e7d0889c814127c0be7bb4ece4385d42a62606704e32e8 (image=172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest, name=ceph-nfs-pacemaker)
Jul 27 14:37:40 dint-sdc-ctrl-03 podman[343693]: 2021-07-27 14:37:40.790747921 +0900 KST m=+0.100014853 container remove 92fa9d5b889b5f9d95e7d0889c814127c0be7bb4ece4385d42a62606704e32e8 (image=172.16.222.212:5000/rhceph/rhceph-4-rhel8:latest, name=ceph-nfs-pacemaker)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Result of monitor operation for ceph-nfs on dint-sdc-ctrl-03: 7 (not running)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Transition 5407 action 19 (ceph-nfs_monitor_60000 on dint-sdc-ctrl-03): expected 'ok' but got 'not running'
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-attrd[107321]: notice: Setting fail-count-ceph-nfs#monitor_60000[dint-sdc-ctrl-03]: 5 -> 6
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-attrd[107321]: notice: Setting last-failure-ceph-nfs#monitor_60000[dint-sdc-ctrl-03]: 1627276358 -> 1627364269
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: warning: Unexpected result (not running) was recorded for monitor of ceph-nfs on dint-sdc-ctrl-03 at Jul 27 14:37:49 2021
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: notice:  * Recover    ceph-nfs                            (                     dint-sdc-ctrl-03 )
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: warning: Unexpected result (not running) was recorded for monitor of ceph-nfs on dint-sdc-ctrl-03 at Jul 27 14:37:49 2021
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-schedulerd[107323]: notice:  * Recover    ceph-nfs                            (                     dint-sdc-ctrl-03 )
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Initiating stop operation ceph-nfs_stop_0 locally on dint-sdc-ctrl-03
Jul 27 14:37:51 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Result of stop operation for ceph-nfs on dint-sdc-ctrl-03: 0 (ok)
Jul 27 14:37:51 dint-sdc-ctrl-03 pacemaker-controld[107324]: notice: Initiating start operation ceph-nfs_start_0 locally on dint-sdc-ctrl-03

- From the pacemaker

Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_process_request)     info: Forwarding cib_modify operation for section status to all (origin=local/crmd/6387)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: Diff: --- 0.193.87 2
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: Diff: +++ 0.193.88 (null)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: +  /cib:  @num_updates=88
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_perform_op)  info: +  /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ceph-nfs']/lrm_rsc_op[@id='ceph-nfs_last_failure_0']:  @transition-key=19:5407:0:97933452-33ab-49ab-b724-071a12dc7ba0, @transition-magic=0:7;19:5407:0:97933452-33ab-49ab-b724-071a12dc7ba0, @call-id=347, @last-rc-change=1627364269
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-based     [107318] (cib_process_request)     info: Completed cib_modify operation for section status: OK (rc=0, origin=dint-sdc-ctrl-03/crmd/6387, version=0.193.88)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld  [107324] (abort_transition_graph)  info: Transition 5504 aborted by operation ceph-nfs_monitor_60000 'modify' on dint-sdc-ctrl-03: Change in recurring result | magic=0:7;19:5407:0:97933452-33ab-49ab-b724-071a12dc7ba0 cib=0.193.88 source=process_graph_event:406 complete=true
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld  [107324] (update_failcount)        info: Updating failcount for ceph-nfs on dint-sdc-ctrl-03 after failed monitor: rc=7 (update=value++, time=1627364269)
Jul 27 14:37:49 dint-sdc-ctrl-03 pacemaker-controld  [107324] (process_graph_event)     notice: Transition 5407 action 19 (ceph-nfs_monitor_60000 on dint-sdc-ctrl-03): expected 'ok' but got 'not running' | target-rc=0 rc=7 call-id=347 event='arrived after initial scheduling'


- From the manila-share.log.1

2021-07-27 14:37:40.626 42 ERROR manila.share.drivers.ganesha.manager [req-566e8125-cfe4-4c37-8f0d-5a388faeaff2 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.AddExport.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server [req-566e8125-cfe4-4c37-8f0d-5a388faeaff2 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 233, in _execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return execute(*args, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 59, in __call__
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return self.execute(*args, **exkwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 101, in execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 474, in add_export
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     "string:EXPORT(Export_Id=%d)" % xid)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 414, in _dbus_send_ganesha
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     message='dbus call %s.%s' % (service, method), **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 242, in _execute
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 187, in wrapped
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 3554, in update_access
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 283, in update_access_rules
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 322, in _update_access_rules
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     share_server=share_server)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/__init__.py", line 308, in update_access
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     self.ganesha.add_export(share['name'], confdict)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 491, in add_export
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5e3bbcae-8ac2-406c-8673-8809bc7cd7b2.conf.Omy0Ne string:EXPORT(Export_Id=1538)
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying\n'
2021-07-27 14:37:40.715 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:38:07.805 42 INFO manila.share.manager [req-87fe4d74-3c50-43bc-9d94-a39aaeb58bbc - - - - -] Updating share status
2021-07-27 14:38:31.990 42 ERROR manila.share.drivers.ganesha.manager [req-907e91a1-85e9-4a9a-8134-4cfeeee45890 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.AddExport.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server [req-907e91a1-85e9-4a9a-8134-4cfeeee45890 9ea9decc0e3c4f56b0f6a63be7809138 b73cd23f3f3a4f6a9b9372a6509c3475 - - -] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-eecb4779-ea86-418b-beb7-5a4651873be2.conf.J1j2w9 string:EXPORT(Export_Id=1539)
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 233, in _execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     return execute(*args, **kwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 59, in __call__
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     return self.execute(*args, **exkwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/utils.py", line 101, in execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-eecb4779-ea86-418b-beb7-5a4651873be2.conf.J1j2w9 string:EXPORT(Export_Id=1539)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 474, in add_export
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     "string:EXPORT(Export_Id=%d)" % xid)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 414, in _dbus_send_ganesha
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     message='dbus call %s.%s' % (service, method), **kwargs)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/manager.py", line 242, in _execute
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server     cmd=e.cmd)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Command: dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-eecb4779-ea86-418b-beb7-5a4651873be2.conf.J1j2w9 string:EXPORT(Export_Id=1539)
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Exit code: 1
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stdout: ''
2021-07-27 14:38:32.141 42 ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n'


Expected results:
- Add normaly more share rules.

Additional info:

- Asked ceph cluster status when hit issue, CU said it wasn't effect to ceph cluster status.

- You can find related data in below

$ ssh <your-id>@supportshell.cee.redhat.com
$ yank 02997902
$ cd 02997902
[yocha@supportshell 02997902]$ ll
total 896932
-rw-rwxrw-+  1 yank  yank     59794 Jul 28 11:44 0010-manila-error.txt
....
-rw-rw-rw-+  1 yank  yank     62488 Jul 29 07:57 0070-systemd-coredump                                 <<== core file. 
drwxrwxrwx+  2 yank  yank      4096 Jul 29 08:34 0080-core.ganesha.tar.gz                              <<== core file.
drwxrwxrwx+ 13 yocha yocha     4096 Jul 28 02:50 sosreport-dint-sdc-ctrl-03-2021-07-28-anuqybu         <<== sosreport from controller node.
drwxrwxrwx+  3 yank  yank        67 Jul 29 00:24 sosreport-dint-sdc-ctrl-03-2021-07-28-anuqybu.tar
....

Let me know if you need addtional info.

Thank you,
Regards,
YoungCheol

Note You need to log in before you can comment on or make changes to this bug.