Bug 1333658

Summary: Fresh lookup for directories must not take place when the redundant #bricks are got down in a disperse volume
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sweta Anandpara <sanandpa>
Component: disperseAssignee: Sunil Kumar Acharya <sheggodu>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: amukherj, aspandey, asrivast, pkarampu, rhinduja, rhs-bugs
Target Milestone: ---   
Target Release: RHGS 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-21 04:28:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1417147    

Description Sweta Anandpara 2016-05-06 06:15:01 UTC 
Description of problem:
------------------------

Found this issue while verifying BZ 1330901. In a replicate volume, if one of the replica bricks is killed, and a directory is accessed from the mountpoint, it does not initiate a fresh lookup for the root/concerned directory, but just reads it from the other pair. Similarly, if the underlying volume type is disperse, it should tolerate killing of that many number of bricks as per the redundancy count. 

This is going to negatively impact performance, when huge data set is involved. 


Version-Release number of selected component (if applicable):
-------------------------------------------------------------

glusterfs-3.7.9-3.el7rhgs.x86_64


How reproducible: Always
-------------------


Steps to Reproduce:
--------------------

1) Create a disperse volume 2*(4+2). Set 'diagnostics.client-log-level' to DEBUG.
2) Mount it over fuse(/nfs) and create a directory say dir1
3) Check for this message "Calling fresh lookup for /dir1"
4) Perform "ls dir1" and again check the no of times you see
the message "Calling fresh lookup on /dir1". We should not see any new 'fresh
lookup' message.
5) Bring down one replica pair, by killing the brick process
6) Perform "ls dir1"
7) Repeat step4 and verify that no 'fresh lookup' message is seen in the logs.
8) Bring down another brick from the same set of 6 bricks. 
9) Repeat step4 and verify that no 'fresh lookup' message is seen in the logs.


Actual results:
----------------

Step7 and Step9 initiates a 'fresh lookup' message for the accessed directory
Expected results:

Expected results:
-----------------
Read xattr must happen from the other bricks that are up, and no fresh lookup must be performed for all directories, after getting brick(s) down.

Additional info:
------------------

[root@dhcp35-210 ~]# gluster v create ec  disperse 6 redundancy 2 10.70.35.210:/bricks/brick0/ec 10.70.35.85:/bricks/brick0/ec 10.70.35.137:/bricks/brick0/ec 10.70.35.13:/bricks/brick0/ec 10.70.35.210:/bricks/brick1/ec 10.70.35.85:/bricks/brick1/ec 10.70.35.137:/bricks/brick1/ec 10.70.35.13:/bricks/brick1/ec 10.70.35.210:/bricks/brick2/ec 10.70.35.85:/bricks/brick2/ec 10.70.35.137:/bricks/brick2/ec 10.70.35.13:/bricks/brick2/ec force
volume create: ec: success: please start the volume to access data
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v info ec
 
Volume Name: ec
Type: Distributed-Disperse
Volume ID: 99ab7b38-f875-4116-936a-5e7e873b9162
Status: Created
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.35.210:/bricks/brick0/ec
Brick2: 10.70.35.85:/bricks/brick0/ec
Brick3: 10.70.35.137:/bricks/brick0/ec
Brick4: 10.70.35.13:/bricks/brick0/ec
Brick5: 10.70.35.210:/bricks/brick1/ec
Brick6: 10.70.35.85:/bricks/brick1/ec
Brick7: 10.70.35.137:/bricks/brick1/ec
Brick8: 10.70.35.13:/bricks/brick1/ec
Brick9: 10.70.35.210:/bricks/brick2/ec
Brick10: 10.70.35.85:/bricks/brick2/ec
Brick11: 10.70.35.137:/bricks/brick2/ec
Brick12: 10.70.35.13:/bricks/brick2/ec
Options Reconfigured:
performance.readdir-ahead: on
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v start ec
volume start: ec: success
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v info ec
 
Volume Name: ec
Type: Distributed-Disperse
Volume ID: 99ab7b38-f875-4116-936a-5e7e873b9162
Status: Started
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.35.210:/bricks/brick0/ec
Brick2: 10.70.35.85:/bricks/brick0/ec
Brick3: 10.70.35.137:/bricks/brick0/ec
Brick4: 10.70.35.13:/bricks/brick0/ec
Brick5: 10.70.35.210:/bricks/brick1/ec
Brick6: 10.70.35.85:/bricks/brick1/ec
Brick7: 10.70.35.137:/bricks/brick1/ec
Brick8: 10.70.35.13:/bricks/brick1/ec
Brick9: 10.70.35.210:/bricks/brick2/ec
Brick10: 10.70.35.85:/bricks/brick2/ec
Brick11: 10.70.35.137:/bricks/brick2/ec
Brick12: 10.70.35.13:/bricks/brick2/ec
Options Reconfigured:
performance.readdir-ahead: on
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        49164     0          Y       26877
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        49159     0          Y       1747 
Brick 10.70.35.13:/bricks/brick0/ec         49159     0          Y       30551
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        49166     0          Y       26915
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v set ec diagnostics.client-log-level DEBUG
volume set: success
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v get ec diagnostics.client-log-level
Option                                  Value                                   
------                                  -----                                   
diagnostics.client-log-level            DEBUG                                   
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# kill -9 26877
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        49159     0          Y       1747 
Brick 10.70.35.13:/bricks/brick0/ec         49159     0          Y       30551
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        49166     0          Y       26915
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        49159     0          Y       1747 
Brick 10.70.35.13:/bricks/brick0/ec         49159     0          Y       30551
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        49166     0          Y       26915
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.13:/bricks/brick0/ec         49159     0          Y       30551
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        49166     0          Y       26915
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.13:/bricks/brick0/ec         N/A       N/A        N       N/A  
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        49166     0          Y       26915
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# kill -9 26915
[root@dhcp35-210 ~]# 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.13:/bricks/brick0/ec         N/A       N/A        N       N/A  
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.13:/bricks/brick0/ec         N/A       N/A        N       N/A  
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        49161     0          Y       1785 
Brick 10.70.35.13:/bricks/brick2/ec         49161     0          Y       30589
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# gluster v status ec
Status of volume: ec
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.210:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick0/ec         49159     0          Y       651  
Brick 10.70.35.137:/bricks/brick0/ec        N/A       N/A        N       N/A  
Brick 10.70.35.13:/bricks/brick0/ec         N/A       N/A        N       N/A  
Brick 10.70.35.210:/bricks/brick1/ec        49165     0          Y       26896
Brick 10.70.35.85:/bricks/brick1/ec         49160     0          Y       671  
Brick 10.70.35.137:/bricks/brick1/ec        49160     0          Y       1766 
Brick 10.70.35.13:/bricks/brick1/ec         49160     0          Y       30570
Brick 10.70.35.210:/bricks/brick2/ec        N/A       N/A        N       N/A  
Brick 10.70.35.85:/bricks/brick2/ec         49161     0          Y       708  
Brick 10.70.35.137:/bricks/brick2/ec        N/A       N/A        N       N/A  
Brick 10.70.35.13:/bricks/brick2/ec         N/A       N/A        N       N/A  
NFS Server on localhost                     2049      0          Y       26935
Self-heal Daemon on localhost               N/A       N/A        Y       26943
NFS Server on 10.70.35.13                   2049      0          Y       30609
Self-heal Daemon on 10.70.35.13             N/A       N/A        Y       30617
NFS Server on 10.70.35.85                   2049      0          Y       729  
Self-heal Daemon on 10.70.35.85             N/A       N/A        Y       737  
NFS Server on 10.70.35.137                  2049      0          Y       1808 
Self-heal Daemon on 10.70.35.137            N/A       N/A        Y       1816 
 
Task Status of Volume ec
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-210 ~]# rpm -qa | grep gluster
glusterfs-libs-3.7.9-3.el7rhgs.x86_64
glusterfs-cli-3.7.9-3.el7rhgs.x86_64
glusterfs-client-xlators-3.7.9-3.el7rhgs.x86_64
glusterfs-fuse-3.7.9-3.el7rhgs.x86_64
glusterfs-geo-replication-3.7.9-3.el7rhgs.x86_64
python-gluster-3.7.5-19.el7rhgs.noarch
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-api-3.7.9-3.el7rhgs.x86_64
glusterfs-rdma-3.7.9-3.el7rhgs.x86_64
gluster-nagios-addons-0.2.6-1.el7rhgs.x86_64
glusterfs-3.7.9-3.el7rhgs.x86_64
glusterfs-server-3.7.9-3.el7rhgs.x86_64
[root@dhcp35-210 ~]# 

======================   CLIENT LOGS   ================================

[root@dhcp35-3 ~]# mkdir /mnt/ec
[root@dhcp35-3 ~]# mount -t glusterfs 10.70.35.210:/ec /mnt/ec
[root@dhcp35-3 ~]# cd /mnt/ec
[root@dhcp35-3 ec]# 
[root@dhcp35-3 ec]# 
[root@dhcp35-3 ec]# 
[root@dhcp35-3 ec]# mkdir abc
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]# ls abc
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]# # After killing one brick
[root@dhcp35-3 ec]# ls abc
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]# # After killing second brick of the same set
[root@dhcp35-3 ec]# ls abc
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:57:48.786160] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]# ls abc
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:57:48.786160] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]# # After killing another brick of different set
[root@dhcp35-3 ec]# ls abc
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:57:48.786160] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 06:00:16.840766] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]# # After killing second brick of the different set
[root@dhcp35-3 ec]# ls abc
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:57:48.786160] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 06:00:16.840766] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 06:02:31.893302] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]#
[root@dhcp35-3 ec]# # After killing third brick
[root@dhcp35-3 ec]# ls abc
ls: cannot access abc: Transport endpoint is not connected
[root@dhcp35-3 ec]# grep -R "Calling fresh lookup for /abc" /var/log/glusterfs/mnt-ec.log 
[2016-05-06 05:02:07.813628] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.819028] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:02:07.825234] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:03:08.494649] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 05:57:48.786160] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 06:00:16.840766] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[2016-05-06 06:02:31.893302] D [MSGID: 0] [dht-common.c:2471:dht_lookup] 0-ec-dht: Calling fresh lookup for /abc on ec-disperse-1
[root@dhcp35-3 ec]# ls
ls: cannot open directory .: Transport endpoint is not connected
[root@dhcp35-3 ec]# 
[root@dhcp35-3 ec]# 
[root@dhcp35-3 ec]# pwd
/mnt/ec
[root@dhcp35-3 ec]#
Comment 3 Sunil Kumar Acharya 2017-01-27 06:35:21 UTC 
Fix for the issue is ported to downstream as part of : https://code.engineering.redhat.com/gerrit/#/c/90692/
Comment 7 Nag Pavan Chilakam 2017-06-19 16:37:11 UTC 
ran above steps mentioned in summary , no fresh lookups are issued, as long as no change is done in the directory view. if there are new files created  post step7 and before step8, that would result in fresh lookups which makes sense


also tried with nl-cache enabled, and worked as expected

version:3.8.4-28

hence moving to verified
Comment 9 errata-xmlrpc 2017-09-21 04:28:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774
Comment 10 errata-xmlrpc 2017-09-21 04:54:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774