Bug 1221470

Summary:	dHT rebalance: Dict_copy log messages when running rebalance on a dist-rep volume
Product:	[Community] GlusterFS	Reporter:	Anuradha <atalur>
Component:	distribute	Assignee:	Anuradha <atalur>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.7.0	CC:	atalur, bugs, gluster-bugs, nbalacha, smohan, spalai
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1220332
Clones:	1227176 (view as bug list)		Environment:
Last Closed:	2015-06-02 08:02:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1220332
Bug Blocks:	1227176

Description Anuradha 2015-05-14 06:31:04 UTC

+++ This bug was initially created as a clone of Bug #1220332 +++

Description of problem:

Lots of messages like the following when running rebalance on a distribute-replicate volume:

[2015-05-11 10:52:54.690469] W [dict.c:1356:dict_copy] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1b6)[0x7fe588263236] (--> /usr/local/lib/libglusterfs.so.0(dict_copy+0x85)[0x7fe58825e3b5] (--> /usr/local/lib/glusterfs/3.8dev/xlator/cluster/replicate.so(afr_lookup_xattr_req_prepare+0x7e)[0x7fe57cfc27c0] (--> /usr/local/lib/glusterfs/3.8dev/xlator/cluster/replicate.so(afr_lookup_do+0x112)[0x7fe57cfc6752] (--> /usr/local/lib/glusterfs/3.8dev/xlator/cluster/replicate.so(afr_lookup+0x2bf)[0x7fe57cfc6f93] ))))) 0-dict: dict is NULL


Version-Release number of selected component (if applicable):


How reproducible:
Everytime

Steps to Reproduce:
1.Create a dist-rep volume and create some files on it
2.Rename the files to trigger file migrations
3. Start rebalance on the volume

Actual results:

Lots of the following type of messages in the rebalance log:

[2015-05-11 10:52:54.690469] W [dict.c:1356:dict_copy] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1b6)[0x7fe588263236] (--> /usr/local/lib/libglusterfs.so.0(dict_copy+0x85)[0x7fe58825e3b5] (--> /usr/local/lib/glusterfs/3.8dev/xlator/cluster/replicate.so(afr_lookup_xattr_req_prepare+0x7e)[0x7fe57cfc27c0] (--> /usr/local/lib/glusterfs/3.8dev/xlator/cluster/replicate.so(afr_lookup_do+0x112)[0x7fe57cfc6752] (--> /usr/local/lib/glusterfs/3.8dev/xlator/cluster/replicate.so(afr_lookup+0x2bf)[0x7fe57cfc6f93] ))))) 0-dict: dict is NULL


Expected results:


Additional info:
No such messages.

--- Additional comment from Anand Avati on 2015-05-12 03:27:53 EDT ---

REVIEW: http://review.gluster.org/10755 (cluster/afr : Do not copy dict when it is NULL) posted (#1) for review on master by Anuradha Talur (atalur)

--- Additional comment from Susant Kumar Palai on 2015-05-12 09:09:35 EDT ---

RCA: __dht_rebalance_create_dst_file does syncop_lookup and does not pass xattr_req. 

   <ret = syncop_lookup (to, loc, &new_stbuf, NULL, NULL, NULL);>

And afr tries to does a dict_copy from this dictionary which is NULL. Hence, the logs. (fixed by http://review.gluster.org/10755)

--- Additional comment from Anand Avati on 2015-05-13 07:54:46 EDT ---

COMMIT: http://review.gluster.org/10755 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 2b92a5c5efc29b05de88631bc5423ae3eec43812
Author: Anuradha <atalur>
Date:   Tue May 12 12:53:30 2015 +0530

    cluster/afr : Do not copy dict when it is NULL
    
    In afr_lookup_xattr_req_prepare(), dict_copy was
    done even though source dict was NULL.
    
    Change-Id: I85a5d2823ba021e7f78c1ce13402a0f16b08cb51
    BUG: 1220332
    Signed-off-by: Anuradha <atalur>
    Reviewed-on: http://review.gluster.org/10755
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Susant Palai <spalai>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 1 Anand Avati 2015-05-14 08:26:57 UTC

REVIEW: http://review.gluster.org/10777 (cluster/afr : Do not copy dict when it is NULL) posted (#2) for review on release-3.7 by Anuradha Talur (atalur)

Comment 2 Anand Avati 2015-05-15 09:10:47 UTC

COMMIT: http://review.gluster.org/10777 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 1562b5af8c8a10e3f2c08b8f16c6db01d3b9f28c
Author: Anuradha <atalur>
Date:   Tue May 12 12:53:30 2015 +0530

    cluster/afr : Do not copy dict when it is NULL
    
           Backport of: http://review.gluster.org/10755
    
    In afr_lookup_xattr_req_prepare(), dict_copy was
    done even though source dict was NULL.
    
    Change-Id: I85a5d2823ba021e7f78c1ce13402a0f16b08cb51
    BUG: 1221470
    Reviewed-on: http://review.gluster.org/10755
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Susant Palai <spalai>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Signed-off-by: Anuradha <atalur>
    Reviewed-on: http://review.gluster.org/10777

Comment 3 Niels de Vos 2015-06-02 08:02:40 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.1, please reopen this bug report.

glusterfs-3.7.1 has been announced on the Gluster Packaging mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.packaging/1
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user