Red Hat Bugzilla – Bug 1400599
[GANESHA] failed to create directory of hostname of new node in var/lib/nfs/ganesha/ in already existing cluster nodes
Last modified: 2017-03-23 01:53:56 EDT
Description of problem: When new node is added in ganesha cluster,directory of the hostname is added to /var/lib/nfs/ganesha/dhcp46-219.lab.eng.blr.redhat.com having simlink to dhcp46-219.lab.eng.blr.redhat.com -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp46-219.lab.eng.blr.redhat.com/nfs/ganesha This directory (having hostname of new node) is missing from all other existing nodes in ganesha cluster after adding new node. ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.Create a 4 node cluster and configure ganesha on it. 2.Add new node to cluster after performing prerequisite for adding new node to cluster [root@dhcp47-3 gdeploy]# /usr/libexec/ganesha/ganesha-ha.sh --add /var/run/gluster/shared_storage/nfs-ganesha/ dhcp46-208.lab.eng.blr.redhat.com 10.70.44.157 Following messages were observed in ganesha.log of all the existing nodes: 01/12/2016 20:13:49 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 01/12/2016 20:13:49 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip 01/12/2016 20:13:49 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha//v4recov) 01/12/2016 20:13:57 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip dhcp46-208.lab.eng.blr.redhat.com 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov) 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip dhcp46-208.lab.eng.blr.redhat.com 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov) 01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2 01/12/2016 20:15:37 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE Actual results: No Directory of hostname is created in /var/lib/nfs/ganesha/ Expected results: Directory of hostname should be created in /var/lib/nfs/ganesha/ No messages regarding Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2 should be observed in ganesha.log Additional info:
upstream mainline patch : http://review.gluster.org/#/c/16036/
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/93172/
Verified this bug on # rpm -qa | grep ganesha nfs-ganesha-2.4.1-3.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.1-3.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-9.el7rhgs.x86_64 Hostname directory of new node is created on already existing nodes. A clean up is required on new node which is being addressed in bug- https://bugzilla.redhat.com/show_bug.cgi?id=1406330 As the issue addressed in this bug is resolved,Hence marking this bug as verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html