Bug 1335505

Summary:	Brick logs spammed with dict_get errors
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ambarish <asoman>
Component:	core	Assignee:	Kotresh HR <khiremat>
Status:	CLOSED ERRATA	QA Contact:	Ambarish <asoman>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	amukherj, asrivast, khiremat, rhinduja, rhs-bugs, storage-qa-internal
Target Milestone:	---	Keywords:	EasyFix, Patch, Regression, ZStream
Target Release:	RHGS 3.1.3
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.9-5	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-06-23 05:23:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1332074
Bug Blocks:	1311817

Description Ambarish 2016-05-12 11:07:43 UTC

Description of problem:
----------------------

I see a looooot of dict_get error logs on my brick logs :

[2016-05-12 10:49:08.219144] I [dict.c:473:dict_get] (-->/usr/lib64/libglusterfs.so.0(default_getxattr_cbk+0x135) [0x7f66e391c015] -->/usr/lib64/glusterfs/3.7.9/xlator/features/marker.so(marker_getxattr_cbk+0xdf) [0x7f66d425d00f] -->/usr/lib64/libglusterfs.so.0(dict_get+0x63) [0x7f66e39016b3] ) 0-dict: !this || key=() [Invalid argument]

[2016-05-12 10:49:08.851229] I [dict.c:473:dict_get] (-->/usr/lib64/libglusterfs.so.0(default_getxattr_cbk+0x135) [0x7f66e391c015] -->/usr/lib64/glusterfs/3.7.9/xlator/features/marker.so(marker_getxattr_cbk+0xdf) [0x7f66d425d00f] -->/usr/lib64/libglusterfs.so.0(dict_get+0x63) [0x7f66e39016b3] ) 0-dict: !this || key=() [Invalid argument]

[2016-05-12 10:49:09.530356] I [dict.c:473:dict_get] (-->/usr/lib64/libglusterfs.so.0(default_getxattr_cbk+0x135) [0x7f66e391c015] -->/usr/lib64/glusterfs/3.7.9/xlator/features/marker.so(marker_getxattr_cbk+0xdf) [0x7f66d425d00f] -->/usr/lib64/libglusterfs.so.0(dict_get+0x63) [0x7f66e39016b3] ) 0-dict: !this || key=() [Invalid argument]

It is being spammed every 5 seconds.

Version-Release number of selected component (if applicable):
------------------------------------------------------------

glusterfs-server-3.7.9-4.el6rhs.x86_64


How reproducible:
----------------

100%

Steps to Reproduce:
--------------------

1. Run Multi threaded self heal.

2. Create new files when self heal is I.P.

3. Check brick logs on any of the nodes.

Actual results:
---------------

A lot of dict_get errors on the bricks.

Expected results:
----------------

Brick logs should not be spammed with dict_get errors


Additional info:
---------------

Setup : 4 servers , 4 clients,2*2 volume

Upstream bug :
https://bugzilla.redhat.com/show_bug.cgi?id=1332074

Comment 5 Niels de Vos 2016-05-12 13:02:09 UTC

I guess that http://review.gluster.org/14144 addresses it. Kotresh?

Comment 6 Kotresh HR 2016-05-16 05:04:31 UTC

Yes, Patch #14144 addresses it. I will back port it to downstream.

Following are upstream patches merged.

Master:
http://review.gluster.org/#/c/13793/

release-3.7:
http://review.gluster.org/#/c/14144/

Comment 7 Atin Mukherjee 2016-05-16 05:14:07 UTC

Since the fix is already available upstream moving the bug to POST state.

Comment 8 Kotresh HR 2016-05-16 11:06:25 UTC

Downstream Patch:
https://code.engineering.redhat.com/gerrit/74369

Comment 11 Ambarish 2016-05-23 14:11:46 UTC

Tested the same scenario on 3.7.9-5,FUSE mount.
Did not observe any dict-get errors/spammed brick logs.
Marking as Verified.

Comment 13 errata-xmlrpc 2016-06-23 05:23:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240