Bug 1400599 - [GANESHA] failed to create directory of hostname of new node in var/lib/nfs/ganesha/ in already existing cluster nodes
Summary: [GANESHA] failed to create directory of hostname of new node in var/lib/nfs/g...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: common-ha
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: ---
: RHGS 3.2.0
Assignee: Kaleb KEITHLEY
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks: 1351528 1400613 1405576 1405577
TreeView+ depends on / blocked
 
Reported: 2016-12-01 15:05 UTC by Manisha Saini
Modified: 2017-03-23 05:53 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.8.4-9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1400613 (view as bug list)
Environment:
Last Closed: 2017-03-23 05:53:56 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 0 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Manisha Saini 2016-12-01 15:05:28 UTC
Description of problem:
When new node is added in ganesha cluster,directory of the hostname is added to /var/lib/nfs/ganesha/dhcp46-219.lab.eng.blr.redhat.com having simlink to 

dhcp46-219.lab.eng.blr.redhat.com -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp46-219.lab.eng.blr.redhat.com/nfs/ganesha

This directory (having hostname of new node) is missing from all other existing nodes in ganesha cluster after adding new node.

ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Create a 4 node cluster and configure ganesha on it.
2.Add new node to cluster after performing prerequisite for adding new node to cluster

[root@dhcp47-3 gdeploy]# /usr/libexec/ganesha/ganesha-ha.sh --add /var/run/gluster/shared_storage/nfs-ganesha/ dhcp46-208.lab.eng.blr.redhat.com 10.70.44.157

Following messages were observed in ganesha.log of all the existing nodes:

01/12/2016 20:13:49 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
01/12/2016 20:13:49 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip 
01/12/2016 20:13:49 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha//v4recov)
01/12/2016 20:13:57 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip dhcp46-208.lab.eng.blr.redhat.com
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov)
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip dhcp46-208.lab.eng.blr.redhat.com
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov)
01/12/2016 20:14:05 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[dbus_heartbeat] nfs4_load_recov_clids_nolock :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2
01/12/2016 20:15:37 : epoch 8cd10000 : dhcp46-241.lab.eng.blr.redhat.com : ganesha.nfsd-12930[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE


Actual results:

No Directory of hostname is created in  /var/lib/nfs/ganesha/ 
 


Expected results:
Directory of hostname should be created in  /var/lib/nfs/ganesha/ 
No messages regarding  Failed to open v4 recovery dir (/var/lib/nfs/ganesha/dhcp46-208.lab.eng.blr.redhat.com/v4recov), errno=2 should be observed in ganesha.log


Additional info:

Comment 4 Atin Mukherjee 2016-12-13 15:28:54 UTC
upstream mainline patch : http://review.gluster.org/#/c/16036/

Comment 7 Atin Mukherjee 2016-12-16 12:53:16 UTC
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/93172/

Comment 9 Manisha Saini 2016-12-20 09:56:15 UTC
Verified this bug on 

# rpm -qa | grep ganesha
nfs-ganesha-2.4.1-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.1-3.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-9.el7rhgs.x86_64

Hostname directory of new node is created on already existing nodes.

A clean up is required on new node which is being addressed in bug-
https://bugzilla.redhat.com/show_bug.cgi?id=1406330

As the issue addressed in this bug is resolved,Hence marking this bug as verified

Comment 11 errata-xmlrpc 2017-03-23 05:53:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.