Description of problem:
posix_compliance test suite fails with latest RHGS and ganesha builds

Version-Release number of selected component (if applicable):
nfs-ganesha-2.4.1-2.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.1-2.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-8.el7rhgs.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Create a nfs-ganesha cluster, create a volume, and enable ganesha on it.
2. Mount the volume on client.
3. Run posix_compliance test suite
   /opt/qa/tools/system_light/run.sh -w /mnt/test_nfs -t posix_compliance -l /var/tmp/posix_realip_1.log

Actual results:

Seen new faiures while running posix_compliance test.
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  72, 76

Expected results:
posix compliance should get passes.

Additional info:

However, the test

/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/open/07.t    (Wstat: 0 Tests: 23 Failed: 3)
  Failed tests:  5, 7, 9

are known failures and remaining ones are new failures.

On v3 mount:
------------

Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  72, 76
Files=185, Tests=1962, 115 wallclock secs ( 1.42 usr  0.56 sys + 16.11 cusr 32.44 csys = 50.53 CPU)
Result: FAIL


On v4 mount:
------------

Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/open/07.t    (Wstat: 0 Tests: 23 Failed: 3)
  Failed tests:  5, 7, 9
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  72, 76
Files=185, Tests=1962, 116 wallclock secs ( 1.47 usr  0.52 sys + 16.04 cusr 32.55 csys = 50.58 CPU)
Result: FAIL


ganesha-gfapi log snippet:
--------------------------
while running link test:
------------------------
[2016-12-13 13:05:31.358073] I [MSGID: 108031] [afr-common.c:2250:afr_local_discovery_cbk] 0-vol2-replicate-0: selecting local read_child vol2-client-0
[2016-12-13 13:05:31.358089] I [MSGID: 108031] [afr-common.c:2250:afr_local_discovery_cbk] 0-vol2-replicate-2: selecting local read_child vol2-client-4
[2016-12-13 13:05:31.358497] I [MSGID: 108031] [afr-common.c:2250:afr_local_discovery_cbk] 0-vol2-replicate-4: selecting local read_child vol2-client-8
[2016-12-13 13:05:35.763156] W [MSGID: 114031] [client-rpc-fops.c:2775:client3_3_link_cbk] 0-vol2-client-2: remote operation failed: (/fstest_0a9998dc3dd5008d8c756b7e17068b73/fstest_531f91ef24de83b4d9e13c1ea3947cd4 -> /fstest_0a9998dc3dd5008d8c756b7e17068b73/fstest_02ba9a3065e58033daf08685d52986c3) [Permission denied]
[2016-12-13 13:05:35.763329] W [MSGID: 114031] [client-rpc-fops.c:2775:client3_3_link_cbk] 0-vol2-client-3: remote operation failed: (/fstest_0a9998dc3dd5008d8c756b7e17068b73/fstest_531f91ef24de83b4d9e13c1ea3947cd4 -> /fstest_0a9998dc3dd5008d8c756b7e17068b73/fstest_02ba9a3065e58033daf08685d52986c3) [Permission denied]

while running rename test:
---------------------------

[2016-12-13 13:07:08.525788] W [MSGID: 114031] [client-rpc-fops.c:2775:client3_3_link_cbk] 0-vol2-client-11: remote operation failed: (/fstest_cead60967afc5a441f444803e5958966/fstest_a1bc0d30db5661a5ab344324de0ca5bd -> /fstest_cead60967afc5a441f444803e5958966/fstest_2717a3f2564782f3027b97bab3e57e9e) [Permission denied]
[2016-12-13 13:07:08.525771] W [MSGID: 114031] [client-rpc-fops.c:2775:client3_3_link_cbk] 0-vol2-client-10: remote operation failed: (/fstest_cead60967afc5a441f444803e5958966/fstest_a1bc0d30db5661a5ab344324de0ca5bd -> /fstest_cead60967afc5a441f444803e5958966/fstest_2717a3f2564782f3027b97bab3e57e9e) [Permission denied]
[2016-12-13 13:07:08.527478] W [MSGID: 114031] [client-rpc-fops.c:630:client3_3_unlink_cbk] 0-vol2-client-7: remote operation failed [Permission denied]
[2016-12-13 13:07:08.527692] W [MSGID: 114031] [client-rpc-fops.c:630:client3_3_unlink_cbk] 0-vol2-client-6: remote operation failed [Permission denied]
[2016-12-13 13:07:08.528243] W [MSGID: 109034] [dht-rename.c:654:dht_rename_unlink_cbk] 0-vol2-dht: /fstest_cead60967afc5a441f444803e5958966/fstest_a1bc0d30db5661a5ab344324de0ca5bd: Rename: unlink on vol2-replicate-3 failed  [Permission denied]
[2016-12-13 13:07:08.584617] E [MSGID: 108008] [afr-transaction.c:2557:afr_write_txn_refresh_done] 0-vol2-replicate-3: Failing SETXATTR on gfid d3f175c3-c755-4d6f-909f-2c78893d098e: split-brain observed.
The message "I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vol2-dht: renaming /fstest_cead60967afc5a441f444803e5958966/fstest_a1bc0d30db5661a5ab344324de0ca5bd (hash=vol2-replicate-5/cache=vol2-replicate-0) => /fstest_cead60967afc5a441f444803e5958966/fstest_2717a3f2564782f3027b97bab3e57e9e (hash=vol2-replicate-3/cache=<nul>)" repeated 2 times between [2016-12-13 13:07:02.471515] and [2016-12-13 13:07:09.619584]
[2016-12-13 13:07:09.624662] W [MSGID: 114031] [client-rpc-fops.c:2711:client3_3_rename_cbk] 0-vol2-client-6: remote operation failed [Permission denied]
[2016-12-13 13:07:09.624877] W [MSGID: 114031] [client-rpc-fops.c:2711:client3_3_rename_cbk] 0-vol2-client-7: remote operation failed [Permission denied]
[2016-12-13 13:07:09.625555] I [MSGID: 109030] [dht-rename.c:111:dht_rename_hashed_dir_cbk] 0-vol2-dht: rename /fstest_cead60967afc5a441f444803e5958966/fstest_a1bc0d30db5661a5ab344324de0ca5bd -> /fstest_cead60967afc5a441f444803e5958966/fstest_2717a3f2564782f3027b97bab3e57e9e on vol2-replicate-3 failed, (gfid = d3f175c3-c755-4d6f-909f-2c78893d098e)  [Permission denied]
[2016-12-13 13:07:09.658207] W [MSGID: 114031] [client-rpc-fops.c:2648:client3_3_readdirp_cbk] 0-vol2-client-6: remote operation failed [No such file or directory]
[2016-12-13 13:07:09.658802] W [MSGID: 114031] [client-rpc-fops.c:2648:client3_3_readdirp_cbk] 0-vol2-client-7: remote operation failed [No such file or directory]
[2016-12-13 13:07:09.665546] W [MSGID: 108019] [afr-lk-common.c:1064:afr_log_entry_locks_failure] 0-vol2-replicate-3: Unable to obtain sufficient blocking entry locks on at least one child while attempting RMDIR on {pgfid:45bf4d22-7110-41c5-966f-b9d5cba41d87, name:fstest_a1bc0d30db5661a5ab344324de0ca5bd}.