Bug 1425865

Description of problem:


Rebalance logs are flooded with 

Version-Release number of selected component (if applicable):


How reproducible:
Every time

Steps to Reproduce:
1. Create a pure distribute volume (vol1) with 3 or more bricks
2. Create several files and directories
3. Run a rebalance
gluster v rebalance vol1 start
4. Check the rebalance log for vol1


Actual results:
The logs are flooded with dict_set messages:

[2017-02-22 14:55:55.493229] W [dict.c:416:dict_set] (-->/usr/local/lib/glusterfs/3.11dev/xlator/cluster/distribute.so(+0x5972a) [0x7f476d0ce72a] -->/usr/local/lib/l        ibglusterfs.so.0(dict_set_int32+0x57) [0x7f477abdb826] -->/usr/local/lib/libglusterfs.so.0(dict_set+0x77) [0x7f477abd86a2] ) 0-dict: !this || !value for key=readdir-filter-directories [Invalid argument]


Expected results:
No such messages should be seen in the rebalance logs.

Additional info:

From gdb:

(gdb) bt
#0  dict_set (this=0x0, key=0x7f01fcbe9b3e "readdir-filter-directories", value=0x7f01e40049d0) at dict.c:414
#1  0x00007f020a6b0826 in dict_set_int32 (this=0x0, key=0x7f01fcbe9b3e "readdir-filter-directories", val=1) at dict.c:1795
#2  0x00007f01fcba372a in dht_opendir (frame=0x7f01e40054f0, this=0x7f01f800d6a0, loc=0x7f01f6ffcf50, fd=0x7f01e4003740, xdata=0x0) at dht-common.c:4897
#3  0x00007f020a7039ae in syncop_opendir (subvol=0x7f01f800d6a0, loc=0x7f01f6ffcf50, fd=0x7f01e4003740, xdata_in=0x0, xdata_out=0x0) at syncop.c:1423
#4  0x00007f01fcb650bf in gf_defrag_process_dir (this=0x7f01f800d6a0, defrag=0x7f01f801a0a0, loc=0x7f01f6ffcf50, migrate_data=0x7f01e8000da0, perrno=0x7f01f6ffcc6c)
    at dht-rebalance.c:2763
#5  0x00007f01fcb6663f in gf_defrag_fix_layout (this=0x7f01f800d6a0, defrag=0x7f01f801a0a0, loc=0x7f01f6ffcf50, fix_layout=0x7f01e8000ac0, migrate_data=0x7f01e8000da0)
    at dht-rebalance.c:3203
#6  0x00007f01fcb68932 in gf_defrag_start_crawl (data=0x7f01f800d6a0) at dht-rebalance.c:3973
#7  0x00007f020a7004bb in synctask_wrap (old_task=0x7f01e4001400) at syncop.c:375
#8  0x00007f0208d9dcf0 in ?? () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()



In the rebalance process: 
conf->readdir_optimize == _gf_true 
conf->defrag != NULL

The rebalance process calls opendir on each directory on the volume.

In dht_opendir:


    for (i = 0; i < conf->local_subvols_cnt; i++) {
            if (conf->readdir_optimize == _gf_true) {
                   if (conf->local_subvols[i] != local->first_up_subvol)
                        ret = dict_set_int32 (local->xattr, GF_READDIR_SKIP_DIRS                                                                       , 1);
                          if (ret)
                                 gf_msg (this->name, GF_LOG_ERROR, 0,                                                         DHT_MSG_DICT_SET_FAILED,  "Failed to set dictionary"                                                     
" value :key = %s, ret:%d",GF_READDIR_SKIP_DIRS, ret);


                        }


At this point, local->xattr is NULL.