Bug 1352926 - gluster volume status <volume> client" isn't showing any information when one of the nodes in a 3-way Distributed-Replicate volume is shut down
Summary: gluster volume status <volume> client" isn't showing any information when one...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: cli
Version: 3.8.0
Hardware: Unspecified
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Atin Mukherjee
QA Contact:
URL:
Whiteboard:
Depends On: 1351732 1351880
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-05 13:53 UTC by Atin Mukherjee
Modified: 2016-08-12 09:46 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.8.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1351880
Environment:
Last Closed: 2016-08-12 09:46:59 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Atin Mukherjee 2016-07-05 13:53:01 UTC
+++ This bug was initially created as a clone of Bug #1351880 +++

+++ This bug was initially created as a clone of Bug #1351732 +++

Description of problem:

3-way Distributed-Replicate volume...

[root@node1 ~]# gluster volume status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
node1:brick1                                49165     0          Y       3665
node2:brick2                                49164     0          Y       17094
node3:brick3                                49154     0          Y       19888
node1:brick4                                49172     0          Y       3670
node2:brick5                                49171     0          Y       17099
node3:brick6                                49155     0          Y       19907
NFS Server on localhost                     2049      0          Y       3679
Self-heal Daemon on localhost               N/A       N/A        Y       3689
NFS Server on node3                         2049      0          Y       19927
Self-heal Daemon on node3                   N/A       N/A        Y       19935
NFS Server on node2                         2049      0          Y       20374
Self-heal Daemon on node2                   N/A       N/A        Y       20382

Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks

[root@node1 ~]#

If the first node in the cluster is shut down, the one where the volume was created and started from, when I try and display on the two remaining nodes information about the clients connected to the volume I don't get any information back.

[root@node2 glusterfs]# gluster volume status vol1 client
Client connections for volume vol1
----------------------------------------------
----------------------------------------------

[root@node3 glusterfs]# gluster volume status vol1 clients
Client connections for volume vol1
----------------------------------------------
----------------------------------------------

As soon as the node (node1) is powered back up and everything is up and running again, the clients are visible.

==========
Version-Release number of selected component (if applicable):

All nodes:

gluster-nagios-addons-0.2.5-1.el7rhgs.x86_64
gluster-nagios-common-0.2.3-1.el7rhgs.noarch
glusterfs-3.7.5-19.el7rhgs.x86_64
glusterfs-api-3.7.5-19.el7rhgs.x86_64
glusterfs-cli-3.7.5-19.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-19.el7rhgs.x86_64
glusterfs-fuse-3.7.5-19.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-19.el7rhgs.x86_64
glusterfs-libs-3.7.5-19.el7rhgs.x86_64
glusterfs-rdma-3.7.5-19.el7rhgs.x86_64
glusterfs-server-3.7.5-19.el7rhgs.x86_64
python-gluster-3.7.5-19.el7rhgs.noarch
samba-vfs-glusterfs-4.2.4-13.el7rhgs.x86_64
vdsm-gluster-4.16.30-1.3.el7rhgs.noarch

==========
How reproducible:

Completely

==========
Steps to Reproduce:

See description

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-06-30 12:55:26 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2016-07-01 01:43:09 EDT ---

RCA:

In CLI side the response dictionary is parsed assuming all the bricks to be up. Since in this case one of the node was brought down client details for the bricks hosted by the same node were not available in the dictionary resulting into a blank output for 'gluster volume status <volname> clients'

--- Additional comment from Vijay Bellur on 2016-07-01 02:21:00 EDT ---

REVIEW: http://review.gluster.org/14842 (cli: print volume status client output for partial bricks) posted (#1) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-05 07:45:40 EDT ---

COMMIT: http://review.gluster.org/14842 committed in master by Jeff Darcy (jdarcy) 
------
commit 427ef5511232614bafcab686ad797cebb6a2d6b5
Author: Atin Mukherjee <amukherj>
Date:   Fri Jul 1 11:47:48 2016 +0530

    cli: print volume status client output for partial bricks
    
    In cli the response dictionary is parsed assuming all the bricks to be up.
    If in a given cluster one of the node is down client details for the
    bricks hosted by the same node are not available in the dictionary resulting
    into a blank output for 'gluster volume status <volname> clients'
    
    Fix is to ignore the ret value for dict_get for those keys.
    
    Change-Id: If4fb65b8807ea3ac71b3ed1a754ea75f599e3613
    BUG: 1351880
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/14842
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>

Comment 1 Vijay Bellur 2016-07-05 13:53:53 UTC
REVIEW: http://review.gluster.org/14865 (cli: print volume status client output for partial bricks) posted (#1) for review on release-3.8 by Atin Mukherjee (amukherj)

Comment 2 Vijay Bellur 2016-07-08 17:18:08 UTC
COMMIT: http://review.gluster.org/14865 committed in release-3.8 by Atin Mukherjee (amukherj) 
------
commit be03507d5224cf44c0e8ca4120c02ce409f5b08c
Author: Atin Mukherjee <amukherj>
Date:   Fri Jul 1 11:47:48 2016 +0530

    cli: print volume status client output for partial bricks
    
    Backport of http://review.gluster.org/14842
    
    In cli the response dictionary is parsed assuming all the bricks to be up.
    If in a given cluster one of the node is down client details for the
    bricks hosted by the same node are not available in the dictionary resulting
    into a blank output for 'gluster volume status <volname> clients'
    
    Fix is to ignore the ret value for dict_get for those keys.
    
    Change-Id: If4fb65b8807ea3ac71b3ed1a754ea75f599e3613
    BUG: 1352926
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/14842
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-on: http://review.gluster.org/14865
    Reviewed-by: Kaushal M <kaushal>

Comment 3 Niels de Vos 2016-08-12 09:46:59 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.2, please open a new bug report.

glusterfs-3.8.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/announce/2016-August/000058.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.