Bug 1210712

Summary: nfs-ganesha: ganesha-ha.sh teardown leaves the /var/lib/nfs symlink as it is.
Product: [Community] GlusterFS Reporter: Saurabh <saujain>
Component: common-haAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: high    
Version: mainlineCC: bugs, gluster-bugs, kkeithle, mzywusko, ndevos, skoduri, vagarwal
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1213927 (view as bug list) Environment:
Last Closed: 2015-10-25 14:55:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1188184, 1213927    

Description Saurabh 2015-04-10 12:13:54 UTC
Description of problem:
ganesha-ha.sh teardown leaves the /var/lib/nfs symlink as it is.
The problem happens is that if you trying to bring up the glusterfs-nfs up, then it won't come up.

Version-Release number of selected component (if applicable):
glusterfs-3.7dev-0.910.git17827de.el6.x86_64
nfs-ganesha-2.2-0.rc7.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. have a 6x2 volume and pre-requisites done for nfs-ganesha HA
2. now try to bring the nfs-ganesha HA
3. bring the HA down using the ganesha-ha.sh --teardown command
4. gluster volume set <volname> nfs.disable off

Actual results:
glusterfs-nfs won't come up
because of this cause,
[root@nfs2 ~]# ls -ltr /var/lib/nfs
nfs         nfs.backup/ 
[root@nfs2 ~]# ls -ltr /var/lib/nfs
lrwxrwxrwx 1 root root 13 Apr  9 12:48 /var/lib/nfs -> /mnt/nfs2/nfs

logs from /var/log/messages
Apr 10 14:27:52 nfs2 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Apr 10 14:28:23 nfs2 rpc.statd[21463]: Version 1.2.3 starting
Apr 10 14:28:23 nfs2 sm-notify[21465]: Version 1.2.3 starting
Apr 10 14:28:23 nfs2 sm-notify[21465]: Failed to open sm: Transport endpoint is not connected
Apr 10 14:28:23 nfs2 sm-notify[21465]: Failed to open directory sm.bak: No such file or directory
Apr 10 14:28:23 nfs2 rpc.statd[21463]: Failed to open directory sm: No such file or directory
Apr 10 14:28:23 nfs2 rpc.statd[21463]: Initializing NSM state
Apr 10 14:28:23 nfs2 rpc.statd[21463]: Failed to create /var/lib/nfs/statd/state.new: No such file or directory
Apr 10 14:28:32 nfs2 rpc.statd[21550]: Version 1.2.3 starting
Apr 10 14:28:32 nfs2 sm-notify[21551]: Version 1.2.3 starting
Apr 10 14:28:32 nfs2 sm-notify[21551]: Failed to open sm: No such file or directory
Apr 10 14:28:32 nfs2 sm-notify[21551]: Failed to open directory sm.bak: No such file or directory
Apr 10 14:28:32 nfs2 rpc.statd[21550]: Failed to open directory sm: No such file or directory
Apr 10 14:28:32 nfs2 rpc.statd[21550]: Initializing NSM state
Apr 10 14:28:32 nfs2 rpc.statd[21550]: Failed to create /var/lib/nfs/statd/state.new: No such file or directory

logs from /var/log/glusterfs/nfs.log
[2015-04-10 08:58:38.107753] I [MSGID: 100030] [glusterfsd.c:2288:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7dev (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/251116e567a53584109c49a8edd8b861.socket)
[2015-04-10 08:58:38.122393] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-04-10 08:58:39.135380] I [rpcsvc.c:2200:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
[2015-04-10 08:58:39.143857] W [mount3.c:3818:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled!
[2015-04-10 08:58:39.146860] W [nlm4.c:2463:nlm4svc_init] 0-nfs-NLM: unable to get pid of rpc.statd from /var/run/rpc.statd.pid 
[2015-04-10 08:58:39.173551] E [nlm4.c:2487:nlm4svc_init] 0-nfs-NLM: unable to start /sbin/rpc.statd
[2015-04-10 08:58:39.173589] E [nfs.c:1409:init] 0-nfs: Failed to initialize protocols
[2015-04-10 08:58:39.173599] E [xlator.c:426:xlator_init] 0-nfs-server: Initialization of volume 'nfs-server' failed, review your volfile again
[2015-04-10 08:58:39.173610] E [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing translator failed
[2015-04-10 08:58:39.173620] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
[2015-04-10 08:58:39.174164] W [glusterfsd.c:1212:cleanup_and_exit] (--> 0-: received signum (0), shutting down


Expected results:
symlink should be removed, only after that the rpc.statd can come up. Only after that the glusterfs-nfs will come up.

Additional info:

Comment 1 Anand Avati 2015-04-21 14:56:20 UTC
REVIEW: http://review.gluster.org/10316 (common-ha: nfs-ganesha: teardown, delete leaves the /var/lib/nfs symlink) posted (#1) for review on master by Kaleb KEITHLEY (kkeithle)

Comment 2 Anand Avati 2015-04-21 16:01:08 UTC
REVIEW: http://review.gluster.org/10316 (common-ha: nfs-ganesha: teardown, delete leaves the /var/lib/nfs symlink) posted (#2) for review on master by Kaleb KEITHLEY (kkeithle)

Comment 3 Anand Avati 2015-04-22 17:03:09 UTC
REVIEW: http://review.gluster.org/10316 (common-ha: nfs-ganesha: teardown, delete leaves the /var/lib/nfs symlink) posted (#3) for review on master by Kaleb KEITHLEY (kkeithle)

Comment 4 Kaleb KEITHLEY 2015-04-22 17:17:27 UTC
*** Bug 1202893 has been marked as a duplicate of this bug. ***

Comment 5 Nagaprasad Sathyanarayana 2015-10-25 14:55:44 UTC
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.