Bug 1570541

Summary: [Ganesha] Ganesha enable command errors out while setting up ganesha on 4 node out of 5 node gluster cluster
Product: Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: nfs-ganeshaAssignee: Jiffin <jthottan>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: unspecified Docs Contact:
Priority: high    
Version: rhgs-3.4CC: dang, ffilz, grajoria, jthottan, rhs-bugs, sankarshan, sheggodu, storage-qa-internal, vdas
Target Milestone: ---Keywords: Regression
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1573078 (view as bug list) Environment:
Last Closed: 2018-09-04 06:46:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1573078    
Bug Blocks: 1503137    

Description Manisha Saini 2018-04-23 07:35:08 UTC
Description of problem:

Ganesha enable command errors out while setting up ganesha on 4 node out of 5 node gluster cluster

# gluster nfs-ganesha enable
Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue?
 (y/n) y
This will take a few minutes to complete. Please wait ..
nfs-ganesha: failed: Commit failed on dhcp37-121.lab.eng.blr.redhat.com. Please check log file for details.


glusterd.log
----------
[2018-04-22 19:01:38.994769] I [MSGID: 106474] [glusterd-ganesha.c:433:check_host_list] 0-management: ganesha host found Hostname is dhcp46-116.lab.eng.blr.redhat.com
[2018-04-22 19:03:34.675747] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Commit failed on dhcp37-121.lab.eng.blr.redhat.com. Please check log file for details.
--------



Version-Release number of selected component (if applicable):

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.5.5-5.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-8.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.5.5-4.el7rhgs.x86_64
nfs-ganesha-2.5.5-5.el7rhgs.x86_64


How reproducible:
2/2


Steps to Reproduce:
1.Create 4 node ganesha cluster out of 5 node gluster cluster


Actual results:
"gluster nfs-ganesha" command errors out.But if check on backend,pcs status output shows cluster is up and running on 4 nodes




Expected results:
"gluster nfs-ganesha" should not error out

Additional info:


# pcs status
Cluster name: ganesha-ha-360
Stack: corosync
Current DC: dhcp47-193.lab.eng.blr.redhat.com (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
Last updated: Mon Apr 23 00:05:48 2018
Last change: Sun Apr 22 23:53:58 2018 by root via cibadmin on dhcp47-193.lab.eng.blr.redhat.com

4 nodes configured
24 resources configured

Online: [ dhcp46-116.lab.eng.blr.redhat.com dhcp46-184.lab.eng.blr.redhat.com dhcp47-193.lab.eng.blr.redhat.com dhcp47-2.lab.eng.blr.redhat.com ]

Full list of resources:

 Clone Set: nfs_setup-clone [nfs_setup]
     Started: [ dhcp46-116.lab.eng.blr.redhat.com dhcp46-184.lab.eng.blr.redhat.com dhcp47-193.lab.eng.blr.redhat.com dhcp47-2.lab.eng.blr.redhat.com ]
 Clone Set: nfs-mon-clone [nfs-mon]
     Started: [ dhcp46-116.lab.eng.blr.redhat.com dhcp46-184.lab.eng.blr.redhat.com dhcp47-193.lab.eng.blr.redhat.com dhcp47-2.lab.eng.blr.redhat.com ]
 Clone Set: nfs-grace-clone [nfs-grace]
     Started: [ dhcp46-116.lab.eng.blr.redhat.com dhcp46-184.lab.eng.blr.redhat.com dhcp47-193.lab.eng.blr.redhat.com dhcp47-2.lab.eng.blr.redhat.com ]
 Resource Group: dhcp47-193.lab.eng.blr.redhat.com-group
     dhcp47-193.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp47-193.lab.eng.blr.redhat.com
     dhcp47-193.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp47-193.lab.eng.blr.redhat.com
     dhcp47-193.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp47-193.lab.eng.blr.redhat.com
 Resource Group: dhcp46-116.lab.eng.blr.redhat.com-group
     dhcp46-116.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp46-116.lab.eng.blr.redhat.com
     dhcp46-116.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp46-116.lab.eng.blr.redhat.com
     dhcp46-116.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp46-116.lab.eng.blr.redhat.com
 Resource Group: dhcp46-184.lab.eng.blr.redhat.com-group
     dhcp46-184.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp46-184.lab.eng.blr.redhat.com
     dhcp46-184.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp46-184.lab.eng.blr.redhat.com
     dhcp46-184.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp46-184.lab.eng.blr.redhat.com
 Resource Group: dhcp47-2.lab.eng.blr.redhat.com-group
     dhcp47-2.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp47-2.lab.eng.blr.redhat.com
     dhcp47-2.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp47-2.lab.eng.blr.redhat.com
     dhcp47-2.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp47-2.lab.eng.blr.redhat.com

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled



-------------------------

Node on which ganesha was not set up---



[2018-04-22 19:03:33.723111] E [MSGID: 106470] [glusterd-ganesha.c:357:glusterd_op_set_ganesha] 0-management: Initial NFS-Ganesha set up failed
[2018-04-22 19:03:33.723136] E [MSGID: 106123] [glusterd-op-sm.c:5959:glusterd_op_ac_commit_op] 0-management: Commit of operation 'Volume (null)' failed: -1
[2018-04-22 19:03:33.821219] E [MSGID: 106062] [glusterd-op-sm.c:4241:glusterd_op_ac_unlock] 0-management: Unable to acquire volname



# cd /etc/ganesha/
[root@dhcp37-121 ganesha]# ll
total 12
-rw-r--r--. 1 root root 1576 Apr 22 07:48 ganesha.conf
-rw-r--r--. 1 root root 2345 Apr 19 10:32 ganesha.conf.example
-rw-r--r--. 1 root root  776 Apr  4 05:32 ganesha-ha.conf.sample

[root@dhcp37-121 ganesha]# cat /var/lib/glusterd/options
global-option-version=90
cluster.enable-shared-storage=enable
nfs-ganesha=disable


# pcs status
Error: cluster is not currently running on this node

[root@dhcp37-121 ganesha]# gluster peer status
Number of Peers: 4

Hostname: dhcp46-116.lab.eng.blr.redhat.com
Uuid: 856cb4ed-c1d1-4da0-b753-b7adaaa46a4f
State: Peer in Cluster (Connected)

Hostname: dhcp47-193.lab.eng.blr.redhat.com
Uuid: 97de04a8-5040-45e0-be11-0b891473d022
State: Peer in Cluster (Connected)

Hostname: dhcp46-184.lab.eng.blr.redhat.com
Uuid: f8c8ba49-c161-4502-a524-54e3b6f995e3
State: Peer in Cluster (Connected)

Hostname: dhcp47-2.lab.eng.blr.redhat.com
Uuid: afd776a4-3ad2-4aab-beec-f4697994be74
State: Peer in Cluster (Connected)

Comment 11 errata-xmlrpc 2018-09-04 06:46:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607

Comment 12 Manisha Saini 2018-09-24 07:00:28 UTC
Setting qe_test_coverage + with no BZ ID,since its been covered as part of every ganesha test case.