Bug 1138395

Summary: Renaming file while rebalance is in progress causes data loss
Product: [Community] GlusterFS Reporter: Shyamsundar <srangana>
Component: distributeAssignee: Shyamsundar <srangana>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: high    
Version: 3.6.0CC: bugs, gluster-bugs, nbalacha, nsathyan, rwheeler, sdharane, shmohan, vbellur, vsomyaju
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.6.0beta1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1130888 Environment:
Last Closed: 2014-11-11 08:38:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 969298, 1130888, 1139998, 1140348, 1146895    
Bug Blocks: 987422, 1117822, 1286059    

Description Shyamsundar 2014-09-04 17:25:12 UTC 
+++ This bug was initially created as a clone of Bug #1130888 +++

Description of problem:
while rebalance is in progress renaming the files causes data loss

Steps to Reproduce:
1.created a 2 brick distribute volume 
2. mounted the volume and created 10000 files
3. add-brick and initiate rebalance, while rebalance is in progress rename all the files
for i in {1..1000}
do
mv $i new$i
done

Actual results:

on the mount point we can see the messages like
[root@localhost mymount]# for i in {1..10000}; do mv $i new$i ; done
mv: cannot move `1747' to `new1747': File exists
mv: cannot move `2485' to `new2485': File exists
mv: cannot move `2577' to `new2577': File exists
mv: cannot move `2586' to `new2586': File exists
mv: cannot move `3455' to `new3455': Structure needs cleaning
mv: cannot move `3626' to `new3626': No such file or directory
mv: cannot move `3829' to `new3829': No such file or directory
mv: cannot move `3830' to `new3830': No such file or directory
mv: cannot move `3944' to `new3944': Structure needs cleaning
mv: cannot move `3963' to `new3963': Structure needs cleaning
mv: cannot move `4180' to `new4180': Structure needs cleaning
mv: cannot move `4506' to `new4506': Structure needs cleaning
mv: cannot move `4591' to `new4591': Structure needs cleaning
mv: cannot move `4601' to `new4601': Structure needs cleaning
mv: cannot move `4602' to `new4602': Structure needs cleaning
mv: cannot move `4611' to `new4611': Structure needs cleaning
mv: cannot move `4644' to `new4644': No such file or directory
mv: cannot move `4709' to `new4709': No such file or directory
mv: cannot move `4814' to `new4814': No such file or directory
mv: cannot move `4835' to `new4835': Structure needs cleaning
mv: cannot move `4852' to `new4852': No such file or directory
mv: cannot move `5009' to `new5009': No such file or directory


the number files has been decreased
[root@localhost mymount]# ls | wc -l
9922


mnt log says 
================
02:06.250800] W [dht-rename.c:482:dht_rename_cbk] 1-dist1-dht: /4852: rename on dist1-client-0 failed (No such file or
 directory)
[2013-05-31 12:02:06.251172] W [fuse-bridge.c:1528:fuse_rename_cbk] 0-glusterfs-fuse: 103658: /4852 -> /new4852 => -1 (No such file o
r directory)
[2013-05-31 12:02:06.314825] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4858 on dist1-client-2 (following 
linkfile) reached link
[2013-05-31 12:02:06.316650] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-2: remote operation failed: File exists. Pat
h: /4858 (00000000-0000-0000-0000-000000000000)
[2013-05-31 12:02:06.930730] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4909 on dist1-client-0 (following 
linkfile) reached link
[2013-05-31 12:02:06.931329] W [dht-common.c:983:dht_lookup_everywhere_cbk] 1-dist1-dht: multiple subvolumes (dist1-client-1 and dist
1-client-0) have file /4909 (preferably rename the file in the backend, and do a fresh lookup)
[2013-05-31 12:02:06.933437] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-0: remote operation failed: File exists. Pat
h: /4909 (00000000-0000-0000-0000-000000000000)
[2013-05-31 12:02:06.977724] W [dht-rename.c:334:dht_rename_unlink_cbk] 1-dist1-dht: /4912: unlink on dist1-client-1 failed (No such 
file or directory)
[2013-05-31 12:02:07.022107] I [dht-common.c:997:dht_lookup_everywhere_cbk] 1-dist1-dht: deleting stale linkfile /4916 on dist1-clien
t-2
[2013-05-31 12:02:07.282241] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4939 on dist1-client-0 (following 
linkfile) reached link
[2013-05-31 12:02:07.283273] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-0: remote operation failed: File exists. Pat
h: /4939 (00000000-0000-0000-0000-000000000000)
[2013-05-31 12:02:07.284610] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4939 on dist1-client-0 (following 
linkfile) reached link
[2013-05-31 12:02:07.288314] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-0: remote operation failed: File exists. Path: /4939 (00000000-0000-0000-0000-000000000000)
[2013-05-31 12:02:07.325669] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4942 on dist1-client-0 (following linkfile) reached link

--- Additional comment from shylesh on 2013-05-31 03:06:30 EDT ---

[root@anshi1 ~]# gluster v info dist1
 
Volume Name: dist1
Type: Distribute
Volume ID: ed55b825-0805-49c8-873c-8447681e687c
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.213:/brick2/dist1
Brick2: 10.70.35.230:/brick2/dist2
Brick3: 10.70.35.213:/brick2/dist3

--- Additional comment from  on 2014-08-11 07:59:23 EDT ---

In rebalance logs, we saw many files migrated from cached to hashed. After migration file should be from data-name by rebalance. But unlinks fails. It
means that rename came in between and caused data loss. 


Race could be something like this:


                            Src-cacehd    Dst-hashed
                                A           A(linkto)

mv A B came and
and A should get renamed
to B at src_cached and
A(linkto) should get
deleted.


Migration migrated the          A(Linkto)       A(data)
file from A to B.          


Rename case                     rename A->B     unlink A


So we are left with renamed linkto file.
Comment 1 Anand Avati 2014-09-04 18:48:05 UTC 
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 2 Anand Avati 2014-09-04 18:48:08 UTC 
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 3 Anand Avati 2014-09-04 18:48:20 UTC 
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 4 Anand Avati 2014-09-04 18:48:27 UTC 
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 5 Anand Avati 2014-09-04 18:48:30 UTC 
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 6 Anand Avati 2014-09-04 18:48:34 UTC 
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 7 Anand Avati 2014-09-04 20:25:37 UTC 
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 8 Anand Avati 2014-09-04 20:25:41 UTC 
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 9 Anand Avati 2014-09-04 20:25:44 UTC 
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 10 Anand Avati 2014-09-04 20:25:51 UTC 
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 11 Anand Avati 2014-09-04 20:25:54 UTC 
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 12 Anand Avati 2014-09-04 20:26:03 UTC 
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 13 Anand Avati 2014-09-05 15:10:43 UTC 
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 14 Anand Avati 2014-09-05 15:10:53 UTC 
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 15 Anand Avati 2014-09-05 15:10:59 UTC 
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 16 Anand Avati 2014-09-05 15:11:17 UTC 
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 17 Anand Avati 2014-09-05 15:11:21 UTC 
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 18 Anand Avati 2014-09-05 15:11:35 UTC 
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 19 Anand Avati 2014-09-05 15:11:47 UTC 
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 20 Anand Avati 2014-09-05 15:19:48 UTC 
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 21 Anand Avati 2014-09-05 15:19:56 UTC 
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 22 Anand Avati 2014-09-05 15:20:00 UTC 
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 23 Anand Avati 2014-09-05 15:20:04 UTC 
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 24 Anand Avati 2014-09-05 15:20:10 UTC 
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 25 Anand Avati 2014-09-05 15:20:14 UTC 
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 26 Anand Avati 2014-09-05 15:20:17 UTC 
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 27 Anand Avati 2014-09-10 15:59:27 UTC 
COMMIT: http://review.gluster.org/8609 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit a325fef8c8f4ca35d65e3891cf72519e925e69f4
Author: Raghavendra G <rgowdapp>
Date:   Thu Sep 4 14:11:14 2014 -0400

    cluster/dht: introduce locking api.
    
    Change-Id: I41389ba91951d3e63e617aa32cd0bee848261c72
    BUG: 1138395
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on-master: http://review.gluster.org/8521
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8609
    Reviewed-by: Jeff Darcy <jdarcy>
Comment 28 Anand Avati 2014-09-10 18:01:08 UTC 
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 29 Anand Avati 2014-09-10 18:01:13 UTC 
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 30 Anand Avati 2014-09-10 18:01:17 UTC 
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 31 Anand Avati 2014-09-10 18:01:20 UTC 
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 32 Anand Avati 2014-09-10 18:01:24 UTC 
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 33 Anand Avati 2014-09-10 18:01:27 UTC 
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 34 Anand Avati 2014-09-12 08:10:47 UTC 
COMMIT: http://review.gluster.org/8610 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit b0dfe8da3fdda31b57177a08de8acec31ac749bf
Author: Raghavendra G <rgowdapp>
Date:   Thu Sep 4 14:12:39 2014 -0400

    libglusterfs/syncop: implement inodelk
    
    Change-Id: Iea489157490b70cb2bb03576b0d4943c6d8f052d
    BUG: 1138395
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on-master: http://review.gluster.org/8522
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8610
Comment 35 Anand Avati 2014-09-12 11:23:30 UTC 
COMMIT: http://review.gluster.org/8611 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit 9ace73a2dd1db0359430a107e9ebc2a0dcb835d2
Author: Raghavendra G <rgowdapp>
Date:   Thu Sep 4 14:14:01 2014 -0400

    cluster/dht: invoke callback when there are no locks to be unlocked.
    
    Change-Id: I375cb68f1075c2d58cf9d09ed6bd5e2746e1637d
    BUG: 1138395
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on-master: http://review.gluster.org/8549
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8611
Comment 36 Anand Avati 2014-09-12 14:06:42 UTC 
COMMIT: http://review.gluster.org/8613 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit 0ae4768d35e76dee30b455e75a4b2552e4abb232
Author: Raghavendra G <rgowdapp>
Date:   Thu Sep 4 14:16:17 2014 -0400

    cluster/dht: synchronize rename and file-migration
    
    Change-Id: I4f243c946f76d440680b651235f925e3d0ebf0fd
    BUG: 1138395
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on-master: http://review.gluster.org/8523
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8613
Comment 37 Anand Avati 2014-09-12 14:07:46 UTC 
COMMIT: http://review.gluster.org/8614 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit fa599ee2de35d4538ae395935be2fbe8184f0806
Author: Shyam <srangana>
Date:   Thu Sep 4 14:17:48 2014 -0400

    cluster/dht: Treat linkto file rename failure as non-critial error
    
    It is a critical failure iff we fail to rename the cached file
    if the rename of the linkto failed, it is not a critical failure,
    and we do not want to lose the created hard link for the new
    name as that could have been read by other clients.
    
    NOTE: If another client is attempting the same oldname -> newname
    rename, and finds both file names as existing, and are hard links
    to each other, then FUSE would send in an unlink for oldname. In
    this time duration if we treat the linkto as a critical error and
    unlink the newname we created, we would have effectively lost the
    file to rename operations.
    
    Repercussions of treating this as a non-critical error is that
    we could leave behind a stale linkto file and/or not create the new
    linkto file, the second case would be rectified by a subsequent
    lookup, the first case by a rebalance, like for all stale linkto
    files
    
    Change-Id: Ia53ad8b43c3cf8f48ef5b43fd1fec4274e807556
    BUG: 1138395
    Signed-off-by: Shyam <srangana>
    Reviewed-on-master: http://review.gluster.org/8563
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8614
Comment 38 Anand Avati 2014-09-16 17:11:58 UTC 
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 39 Anand Avati 2014-09-16 17:12:01 UTC 
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#6) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 40 Anand Avati 2014-09-17 04:23:19 UTC 
COMMIT: http://review.gluster.org/8615 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit a6123ebfdb67e71eab859f94fb14660e9e3701c6
Author: Shyam <srangana>
Date:   Thu Sep 4 14:21:10 2014 -0400

    cluster/dht: Rename should not fail post hardlink creation
    
    In the rename path, we wind the creation of newname hardlink and
    linkto file in dst hashed a the same time. If the linkto creation
    fails, but the link creation succeeds, we enter the failure code
    and cleanup the created newname hardlink.
    
    In the interim if another client looks up newname and finds it as
    a hardlink from FUSE, it could send an unlink for oldname instead
    of a rename. This combined with the above cleanup code could end
    up losing all the files copies, and thereby losing data.
    
    This fix separates these steps into 2 parts, creating the linkto
    first and then the link file, so that post link file creation no
    failures would cleanup the newname file. If linkto fails then link
    is not attempted, thereby not polluting the name space with
    newname.
    
    Change-Id: I61da8e906060da16a31ea1076eec2f01fd617f44
    BUG: 1138395
    Signed-off-by: Shyam <srangana>
    Reviewed-on-master: http://review.gluster.org/8570
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8615
Comment 41 Anand Avati 2014-09-17 14:21:46 UTC 
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 42 Anand Avati 2014-09-18 17:50:42 UTC 
COMMIT: http://review.gluster.org/8626 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit 11485955083762d1c266dfae728a1abfd27aea63
Author: Nithya Balachandran <nbalacha>
Date:   Fri Sep 5 10:50:51 2014 -0400

    Cluster/DHT: Changing rename log severity
    
    Changing log level for a rename message from debug
    to info to improve debuggability
    
    Change-Id: I53031fcf97fffd62095692477330ecde0cf47dcd
    BUG: 1138395
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on-master: http://review.gluster.org/8582
    Reviewed-by: Vijay Bellur <vbellur>
    Tested-by: Gluster Build System <jenkins.com>
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/8626
Comment 43 Anand Avati 2014-09-19 14:05:13 UTC 
REVIEW: http://review.gluster.org/8782 (syncop: Invoke dict_unref() in inodelk only if dictionary is not NULL) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
Comment 44 Anand Avati 2014-09-20 08:54:59 UTC 
COMMIT: http://review.gluster.org/8782 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit 10d45b9783b751c1f78b670281fd1d57ae377642
Author: Vijay Bellur <vbellur>
Date:   Fri Sep 19 10:04:22 2014 -0400

    syncop: Invoke dict_unref() in inodelk only if dictionary is not NULL
    
    In the absence of this check, logs can get flooded with messages like this when rebalance is run:
    
    [2014-09-04 17:48:07.148262] W [dict.c:480:dict_unref] (-->/lib64/libc.so.6()
    [0x30daa47a00] (-->/usr/local/lib/libglusterfs.so.0(synctask_wrap+0x12)
    [0x7fa20b7c6ec2]
    (-->/usr/local/lib/glusterfs/3.7dev/xlator/cluster/distribute.so(dht_migrate_file+0x23f)
    [0x7fa200fdb58f]))) 0-dict: dict is NULL
    
    Change-Id: I4c93e4485293b35d86ba07df4d583d2758ec3f49
    BUG: 1138395
    Signed-off-by: Vijay Bellur <vbellur>
    Reviewed-on-master: http://review.gluster.org/8601
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-on: http://review.gluster.org/8782
Comment 45 Niels de Vos 2014-09-22 12:45:45 UTC 
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/
Comment 46 Niels de Vos 2014-11-11 08:38:14 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users