Bug 1001432

Summary:	quota build 3: glusterd.vol.log reports "E" messages
Product:	Red Hat Gluster Storage	Reporter:	Saurabh <saujain>
Component:	glusterd	Assignee:	Krutika Dhananjay <kdhananj>
Status:	CLOSED ERRATA	QA Contact:	Saurabh <saujain>
Severity:	medium	Docs Contact:
Priority:	high
Version:	2.1	CC:	grajaiya, kdhananj, kparthas, mzywusko, rhs-bugs, saujain, vbellur
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0.34rhs	Doc Type:	Bug Fix
Doc Text:	Cause: These log messages result from glusterd doing generic aggregation of responses from peers, rather than using aggregation logic that is specific to a particular quota sub-command. Consequence: As a result of the above, glusterd used to report the log messages reported in the bug. Fix: Now, glusterd does command-specific aggregation of peer responses Result: These log messages will not be seen when the command is successful.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-11-27 15:33:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Saurabh 2013-08-27 04:05:43 UTC

Description of problem:

in nfs.log I can see some error messages related to quota, altough from client point of view I didn't much of a problem

[2013-08-27 03:29:00.780392] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:00.780476] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:00.781227] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:00.781280] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:00.782369] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:00.782421] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:24.205788] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:24.205848] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:24.206172] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:24.206211] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:24.209218] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:24.209258] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:38.741403] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:38.741499] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:38.744267] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:38.744319] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:38.747010] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:38.747056] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[root@rhsauto034 ~]# tail -f /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
[2013-08-27 03:29:24.206172] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:24.206211] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:24.209218] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:24.209258] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:38.741403] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:38.741499] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:38.744267] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:38.744319] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick
[2013-08-27 03:29:38.747010] E [glusterd-utils.c:7487:glusterd_volume_quota_copy_to_op_ctx_dict] 0-management: Failed to get the count of gfids from the rsp dict
[2013-08-27 03:29:38.747056] E [glusterd-syncop.c:721:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick


as per volume status all the nodes were up and running.

Since there is difference in timestamps that we display in logs and in the system, hence it is difficult to make sure which test might have caused this issue.

As of now I can say that I executed fs-sanity overnight and again a test to check the directory level quota.

Version-Release number of selected component (if applicable):

glusterfs-fuse-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-libs-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-api-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-server-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-3.4.0.20rhsquota5-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.20rhsquota5-1.el6rhs.x86_64

How reproducible:
seen some times

Comment 5 Krutika Dhananjay 2013-08-29 09:59:23 UTC

Could you provide the logs of the other nodes in the cluster? By other nodes, I mean the non-originator nodes.

Comment 8 errata-xmlrpc 2013-11-27 15:33:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html