Bug 1386811

Created attachment 1212207 [details]
nfstest_pnfs_20161019101346_3.cap

172.16.8.118 was acting as both MDS and DS.

172.16.8.119 is DS only.

Proposed fix:  https://review.gerrithub.io/298883

With the above patch applied, the issue mentioned earlier is fixed, but ran into a different issue now .

In case if MDS == DS and LAYOUT_COMMIT is sent, file size is not updated.

Logs from the testsuite - 
    FAIL: LAYOUTCOMMIT should not be sent to MDS when DS == MDS  <<< this is client-side issue
    PASS: LAYOUTCOMMIT should be sent to MDS with correct file range
    PASS: LAYOUTCOMMIT should use the layout stateid
    PASS: LAYOUTCOMMIT new offset should be set
    PASS: LAYOUTCOMMIT last write offset (65535) should be one less than the file size (65536)
    PASS: LAYOUTCOMMIT layout type should be LAYOUT4_NFSV4_1_FILES
    PASS: LAYOUTCOMMIT layout update field should be empty for LAYOUT4_NFSV4_1_FILES
    PASS: GETATTR asking for file size is sent within LAYOUTCOMMIT compound
    PASS: LAYOUTCOMMIT reply file size changed is not set (ERRATA)
    FAIL: GETATTR should return correct file size within LAYOUTCOMMIT compound, expecting 65536, got 0

Above tests are performed using RHEL-7.3 client. I tried with fedora client. LAYOUTCOMMIT is not sent if MDS is the DS for the given file.

The issue can be reproduced with below steps as well.

file2 is present on a different machine (DS) - (SIZE rightly updated)

[skoduri@skoduri ~]$ ls -ltr /mnt/skoduri/file2
-rw-rw-r--. 1 skoduri skoduri 6 Oct 20 03:05 /mnt/skoduri/file2
[skoduri@skoduri ~]$ echo "hello world" > /mnt/skoduri/file2
[skoduri@skoduri ~]$ ls -ltr /mnt/skoduri/file2
-rw-rw-r--. 1 skoduri skoduri 12 Oct 20 03:05 /mnt/skoduri/file2
[skoduri@skoduri ~]$ 


file3 is present on MDS server -

[skoduri@skoduri ~]$ ls -ltr /mnt/skoduri/file3
-rw-rw-r--. 1 skoduri skoduri 12 Oct 20 03:04 /mnt/skoduri/file3
[skoduri@skoduri ~]$ 
[skoduri@skoduri ~]$ echo "hello" > /mnt/skoduri/file3
[skoduri@skoduri ~]$ ls -ltr /mnt/skoduri/file3
-rw-rw-r--. 1 skoduri skoduri 0 Oct 20 03:06 /mnt/skoduri/file3
[skoduri@skoduri ~]$ 

>>>> File size is shown as '0'. 

[skoduri@skoduri ~]$ sleep 65
[skoduri@skoduri ~]$ ls -ltr /mnt/skoduri/file3
-rw-rw-r--. 1 skoduri skoduri 6 Oct 20 03:06 /mnt/skoduri/file3
[skoduri@skoduri ~]$ 

>>> After certain time filesize is rightly updated.

Since MDS is the DS in this case, there is no cache-invalidation involved. I even disabled gluster md-cache. Maybe some where in ganesha layers itself, we are caching invalid size.

Proposed fix; https://review.gerrithub.io/298963

Another issue reported with invalid size returned post WRITE incase of DS == MDS. There is no LAYOUT_COMMIT sent here. 

Proposed fix: https://review.gerrithub.io/#/c/299035

Above mentioned patches fixed most of the issues except for one case where in if DS != MDS and there is WRITE followed by GETATTR, the size returned is invalid. The reason could be that there could have been delay in the upcall(cache-invalidation) request sent by Gluster and hence the attrs(size) may not have been updated on MDS by then.

The issues mentioned in the bug are addressed except for the last one caused by delay in upcall notification. Since there isnt much to be done for that, closing this bug.