Bug 1401376

+++ This bug was initially created as a clone of Bug #1392713 +++

+++ This bug was initially created as a clone of Bug #1384070 +++

Description of problem:
=======================
I have enabled md-cache with private build available at http://etherpad.corp.redhat.com/md-cache-3-2

I created a 2x2 volume and mounted it on two different clients:
From one client i created a 3GB txt file and triggered about 10Million hardlink creation to it as below
(file mnt-distrep.log is the original txt file)
for i in {1..10000000};do ln mnt-distrep.log hardlink.$i;done

I had the volume mounted on another client, say client2:

From the client2 , I was accessing files using ll or ls -l, everything was fine.
I then chnaged file permissions of one of the hardlink files say hardlink.90 to chmod 0777 hardlink.90(all this while the hardlink creations were still going on on client1)

Now if I did an ls -l, I saw that some of the hardlink files(which probably got created recently) were having the sticky bit ---------T 

I saw that in the 2x2 volume, all files were created in say one replica pair while the ---------T  files for each file was created on the other replica pair(this is due to the cached and hashed concept of dht ie data and link to files)

I then chose one file which was displaying  ----T  on the client2 and kept doing ll or ls -l
It was inconsistently toggling b/w data file and link to file as below


[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21860 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21866 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21869 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43729 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21880 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43748 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21888 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43758 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43764 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43768 root root 3156080712 Oct 12 17:29 hardlink.43363



My understanding is that , in this case instead of caching or looking up from cache it is trying to fetch the file info from the bricks and it  is inconsistently fetching from the hashed brick which is not right


Also, If I stopped the hardlink creation on the first client(by ctrl+c of the first client command line)
I see now consistently displaying only the right file permission as below

[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21860 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21866 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21869 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43729 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21880 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43748 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
---------T. 21888 root root 0 Oct 12 17:33 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43758 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43764 root root 3156080712 Oct 12 17:29 hardlink.43363
[root@dhcp35-180 ]# ll hardlink.43363
-rwxrwxrwx. 43768 root root 3156080712 Oct 12 17:29 hardlink.43363



Version-Release number of selected component (if applicable):
====
as in etherpad


How reproducible:


Steps to Reproduce:
1.create md-cache setup on a 2x2 volume
2. mount volume on two clients
3. from cleint 1: create a file and start create of hardlinks to the file in a loop of say some 1lakh hardlinks
4. From another client , say client2 change file permissions of a hardlink already created say hardlink.10
5. as long as the 1lakh hardlink creation is in progress, keep issuing ls -l of hardlink.20 (some hardlink which has been created)
You will see that the file permissions is sometimes shown as what is expected and sometimes with just the sticky bit

Actual results:


Expected results:

--- Additional comment from Worker Ant on 2016-11-08 00:07:26 EST ---

REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file is made a linkto file) posted (#1) for review on master by Poornima G (pgurusid)

--- Additional comment from Worker Ant on 2016-11-18 03:59:10 EST ---

REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file is made a linkto file) posted (#2) for review on master by Poornima G (pgurusid)

--- Additional comment from Worker Ant on 2016-11-24 04:12:55 EST ---

REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file is made a linkto file) posted (#3) for review on master by Poornima G (pgurusid)

--- Additional comment from Worker Ant on 2016-11-24 04:15:40 EST ---

REVIEW: http://review.gluster.org/15789 (upcall: Do not invalidate if the file is made a linkto file) posted (#4) for review on master by Poornima G (pgurusid)

--- Additional comment from Worker Ant on 2016-11-30 04:29:40 EST ---

REVIEW: http://review.gluster.org/15789 (dht/md-cache: Filter invalidate if the file is made a linkto file) posted (#5) for review on master by Poornima G (pgurusid)

--- Additional comment from Worker Ant on 2016-11-30 04:32:58 EST ---

REVIEW: http://review.gluster.org/15789 (dht/md-cache: Filter invalidate if the file is made a linkto file) posted (#6) for review on master by Poornima G (pgurusid)

--- Additional comment from Worker Ant on 2016-12-02 05:47:02 EST ---

COMMIT: http://review.gluster.org/15789 committed in master by Rajesh Joseph (rjoseph) 
------
commit 4536f7bdf16f8286d67598eda9a46c029f0c0bf4
Author: Poornima G <pgurusid>
Date:   Tue Nov 8 10:32:29 2016 +0530

    dht/md-cache: Filter invalidate if the file is made a linkto file
    
    Upcall as a part of setattr, sends an invalidation and the
    invalidation carries the resulting stat value. When a file
    is converted to linkto files, even then an invalidation
    is set and as a result the mountpoint shows the sticky
    bit in the stat of the file.
    eg: ---------T. 945 root root 0 Nov  8 10:14 hardlink.999
    
    Fix:
    When dht recieves a notification of sticky bit change, it updates
    the flag, to indicate md-cache to send the subsequent lookup.
    
    Change-Id: Ic2fd7a5b196db0754f9b97072e644e6bf69da606
    BUG: 1392713
    Signed-off-by: Poornima G <pgurusid>
    Reviewed-on: http://review.gluster.org/15789
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Niels de Vos <ndevos>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Susant Palai <spalai>
    Reviewed-by: Rajesh Joseph <rjoseph>