Bug 765285 (GLUSTER-3553)

Summary:	'profile info' does not fetch the statistics from other nodes in the cluster
Product:	[Community] GlusterFS	Reporter:	M S Vishwanath Bhat <vbhat>
Component:	unclassified	Assignee:	krishnan parthasarathi <kparthas>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	pre-release	CC:	amarts, gluster-bugs, mzywusko, nsathyan
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:	3.2.5qa3	Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description M S Vishwanath Bhat 2011-09-14 14:23:14 UTC

Created 2 way distribute volume with three machines in the cluster.

 When I ran the 'profile info' on the machine which has the bricks, only profile info of bricks from that machine is displayed. When ran on node which doesn't have a brick but part of the cluster, I get following message.

root@ubuntu2:/mnt# gluster volume profile hosdu info
Cannot reach bricks. Bricks are down

When all the bricks of the volume are on the  same machine profile info works fine.

I see following errors and warnings in the brick logs.

[2011-09-14 19:44:13.485647] D [glusterd-utils.c:3721:glusterd_remove_pending_entry] 0-: returning 0
[2011-09-14 19:44:13.485721] E [glusterd-op-sm.c:2431:glusterd_profile_volume_brick_rsp] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee) [0x7f9fd8f9fd8e] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/gluste
rd.so(+0x35781) [0x7f9fd8fa0781] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_handle_brick_rsp+0xc9) [0x7f9fd8fa05c9]))) 0-: Assertion failed: op_ctx
[2011-09-14 19:44:13.485778] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(+0x35781) [0x7f9fd8fa0781] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_handle_brick_rsp+0xc9) [0x7f9fd8
fa05c9] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x17e) [0x7f9fd8fa02ce]))) 0-dict: !this || !value for key=1-brick
[2011-09-14 19:44:13.485835] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1-cumulative
[2011-09-14 19:44:13.485887] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-duration
[2011-09-14 19:44:13.485940] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-total-read
[2011-09-14 19:44:13.485992] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-total-write
[2011-09-14 19:44:13.486043] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-write-131072
[2011-09-14 19:44:13.486095] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-13-hits
[2011-09-14 19:44:13.486147] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-13-avglatency
[2011-09-14 19:44:13.486208] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-13-minlatency
[2011-09-14 19:44:13.486261] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-13-maxlatency
[2011-09-14 19:44:13.486313] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-14-hits
[2011-09-14 19:44:13.486364] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-14-avglatency
[2011-09-14 19:44:13.486416] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-14-minlatency
[2011-09-14 19:44:13.486467] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-14-maxlatency
[2011-09-14 19:44:13.486519] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-15-hits
[2011-09-14 19:44:13.486571] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-15-avglatency
[2011-09-14 19:44:13.486637] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-15-minlatency
[2011-09-14 19:44:13.486717] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-15-maxlatency
[2011-09-14 19:44:13.486782] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17
a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-23-hits
[2011-09-14 19:44:13.486848] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-23-avglatency
[2011-09-14 19:44:13.486906] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-23-minlatency
[2011-09-14 19:44:13.486976] W [dict.c:314:dict_set] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_profile_volume_brick_rsp+0x19f) [0x7f9fd8fa02ef] (-->/usr/local/lib/libglusterfs.so.0(dict_foreach+0x3d) [0x7f9fdbc17a6d] (-->/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(_profile_volume_add_brick_rsp+0x8a) [0x7f9fd8f9d90a]))) 0-dict: !this || !value for key=1--1-23-maxlatency

Comment 1 Anand Avati 2011-09-16 02:06:32 UTC

CHANGE: http://review.gluster.com/430 (If there are no bricks of a volume running 'local' to glusterd) merged in master by Vijay Bellur (vijay)

Comment 2 Anand Avati 2011-10-28 12:08:16 UTC

CHANGE: http://review.gluster.com/641 ("count" key is not set if no brick in the volume (located across peers) is) merged in release-3.2 by Vijay Bellur (vijay)

Comment 3 M S Vishwanath Bhat 2011-10-31 09:19:58 UTC

Checked with the 3.2.5qa3 and now the profile info will give the info about bricks which is managed by other glusterd's.

Comment 4 Anand Avati 2011-11-18 08:02:02 UTC

CHANGE: http://review.gluster.com/642 ("count" key is not set if no brick in the volume (located across peers) is) merged in master by Vijay Bellur (vijay)