Bug 1351880

Summary: gluster volume status <volume> client" isn't showing any information when one of the nodes in a 3-way Distributed-Replicate volume is shut down
Product: [Community] GlusterFS Reporter: Atin Mukherjee <amukherj>
Component: cliAssignee: Atin Mukherjee <amukherj>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: unspecified    
Version: mainlineCC: bugs, ccalhoun, storage-qa-internal
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.9.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1351732
: 1352926 (view as bug list) Environment:
Last Closed: 2017-03-27 18:15:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1351732    
Bug Blocks: 1352926    

Description Atin Mukherjee 2016-07-01 06:19:48 UTC
+++ This bug was initially created as a clone of Bug #1351732 +++

Description of problem:

3-way Distributed-Replicate volume...

[root@node1 ~]# gluster volume status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
node1:brick1                                49165     0          Y       3665
node2:brick2                                49164     0          Y       17094
node3:brick3                                49154     0          Y       19888
node1:brick4                                49172     0          Y       3670
node2:brick5                                49171     0          Y       17099
node3:brick6                                49155     0          Y       19907
NFS Server on localhost                     2049      0          Y       3679
Self-heal Daemon on localhost               N/A       N/A        Y       3689
NFS Server on node3                         2049      0          Y       19927
Self-heal Daemon on node3                   N/A       N/A        Y       19935
NFS Server on node2                         2049      0          Y       20374
Self-heal Daemon on node2                   N/A       N/A        Y       20382

Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks

[root@node1 ~]#

If the first node in the cluster is shut down, the one where the volume was created and started from, when I try and display on the two remaining nodes information about the clients connected to the volume I don't get any information back.

[root@node2 glusterfs]# gluster volume status vol1 client
Client connections for volume vol1
----------------------------------------------
----------------------------------------------

[root@node3 glusterfs]# gluster volume status vol1 clients
Client connections for volume vol1
----------------------------------------------
----------------------------------------------

As soon as the node (node1) is powered back up and everything is up and running again, the clients are visible.

==========
Version-Release number of selected component (if applicable):

All nodes:

gluster-nagios-addons-0.2.5-1.el7rhgs.x86_64
gluster-nagios-common-0.2.3-1.el7rhgs.noarch
glusterfs-3.7.5-19.el7rhgs.x86_64
glusterfs-api-3.7.5-19.el7rhgs.x86_64
glusterfs-cli-3.7.5-19.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-19.el7rhgs.x86_64
glusterfs-fuse-3.7.5-19.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-19.el7rhgs.x86_64
glusterfs-libs-3.7.5-19.el7rhgs.x86_64
glusterfs-rdma-3.7.5-19.el7rhgs.x86_64
glusterfs-server-3.7.5-19.el7rhgs.x86_64
python-gluster-3.7.5-19.el7rhgs.noarch
samba-vfs-glusterfs-4.2.4-13.el7rhgs.x86_64
vdsm-gluster-4.16.30-1.3.el7rhgs.noarch

==========
How reproducible:

Completely

==========
Steps to Reproduce:

See description

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-06-30 12:55:26 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2016-07-01 01:43:09 EDT ---

RCA:

In CLI side the response dictionary is parsed assuming all the bricks to be up. Since in this case one of the node was brought down client details for the bricks hosted by the same node were not available in the dictionary resulting into a blank output for 'gluster volume status <volname> clients'

Comment 1 Vijay Bellur 2016-07-01 06:21:00 UTC
REVIEW: http://review.gluster.org/14842 (cli: print volume status client output for partial bricks) posted (#1) for review on master by Atin Mukherjee (amukherj)

Comment 2 Vijay Bellur 2016-07-05 11:45:40 UTC
COMMIT: http://review.gluster.org/14842 committed in master by Jeff Darcy (jdarcy) 
------
commit 427ef5511232614bafcab686ad797cebb6a2d6b5
Author: Atin Mukherjee <amukherj>
Date:   Fri Jul 1 11:47:48 2016 +0530

    cli: print volume status client output for partial bricks
    
    In cli the response dictionary is parsed assuming all the bricks to be up.
    If in a given cluster one of the node is down client details for the
    bricks hosted by the same node are not available in the dictionary resulting
    into a blank output for 'gluster volume status <volname> clients'
    
    Fix is to ignore the ret value for dict_get for those keys.
    
    Change-Id: If4fb65b8807ea3ac71b3ed1a754ea75f599e3613
    BUG: 1351880
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/14842
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>

Comment 3 Shyamsundar 2017-03-27 18:15:34 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report.

glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html
[2] https://www.gluster.org/pipermail/gluster-users/