Bug 1247450

Summary:	NFS-Ganesha: Server becomes unresponsive intermittently when adding an export
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Meghana <mmadhusu>
Component:	nfs-ganesha	Assignee:	Soumya Koduri <skoduri>
Status:	CLOSED CURRENTRELEASE	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	akhakhar, jthottan, kkeithle, ndevos, nlevinki, pasik, sankarshan, skoduri
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-09 12:29:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1330365
Bug Blocks:

Description Meghana 2015-07-28 03:50:56 UTC

Description of problem:
NFS-Ganesha becomes unresponsive.


Version-Release number of selected component (if applicable):


How reproducible:
Intermittently.


Steps to Reproduce:
1.
2.
3.

Actual results:
dbus-send to add export,
Error org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

/var/log/ganesha.log

27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (2:2)
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
27/07/2015 20:08:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
27/07/2015 20:09:59 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
27/07/2015 20:10:37 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
27/07/2015 20:32:27 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy.  Not sending heartbeat
27/07/2015 20:49:50 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
27/07/2015 20:53:16 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
27/07/2015 20:53:34 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[work-8] mnt_Mnt :NFS3 :EVENT :MOUNT: Performance warning: Export entry is not cached
27/07/2015 21:02:10 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
27/07/2015 21:09:35 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
27/07/2015 21:51:48 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
27/07/2015 22:33:22 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy.  Not sending heartbeat
27/07/2015 22:40:13 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy.  Not sending heartbeat
27/07/2015 22:41:37 : epoch 55b64282 : vm1 : ganesha.nfsd-13753[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy.  Not sending heartbeat

/var/log/messages

Jul 27 22:24:02 vm1 pengine[1015]:  warning: unpack_rsc_op_failure: Processing failed op monitor for nfs-mon:1 on vm1: unknown error (1)




Expected results:
NFS-Ganesha should not enter this state.


Additional info:

Comment 2 Meghana 2015-07-28 03:54:12 UTC

Apeksha, even though it is not reproducible 100%, just add the exact steps that you had performed before you hit this state.

Comment 4 Soumya Koduri 2016-01-28 11:17:38 UTC

This could be related to bug1301450 where in refresh-config (AddExport) failed as ganesha dbus thread went into infinite loop during unexport.

Comment 5 Niels de Vos 2016-06-20 12:05:51 UTC

Not a duplicate of bug 1335114, this is still the case with adding an export.

Comment 7 Red Hat Bugzilla 2023-09-14 03:02:40 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days