Description of problem: while running posix compliance test suit on ganesha mount, ftruncate/00.t and open/00.t tests are failing. ----- 3.4 ----- Test Summary Report ------------------- /home/ntfs-3g-pjd-fstest/tests/chmod/00.t (Wstat: 0 Tests: 106 Failed: 2) Failed tests: 31, 39 /home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t (Wstat: 0 Tests: 26 Failed: 1) Failed test: 24 /home/ntfs-3g-pjd-fstest/tests/mknod/03.t (Wstat: 0 Tests: 12 Failed: 9) Failed tests: 1-3, 5-7, 9-11 /home/ntfs-3g-pjd-fstest/tests/open/00.t (Wstat: 0 Tests: 47 Failed: 5) Failed tests: 30-34 /home/ntfs-3g-pjd-fstest/tests/open/07.t (Wstat: 0 Tests: 23 Failed: 3) Failed tests: 5, 7, 9 Files=210, Tests=2328, 92 wallclock secs ( 0.84 usr 0.29 sys + 10.93 cusr 16.82 csys = 28.88 CPU) Result: FAIL ----- 3.3.1 ----- Test Summary Report ------------------- /home/ntfs-3g-pjd-fstest/tests/chmod/00.t (Wstat: 0 Tests: 106 Failed: 2) Failed tests: 31, 39 /home/ntfs-3g-pjd-fstest/tests/mknod/03.t (Wstat: 0 Tests: 12 Failed: 9) Failed tests: 1-3, 5-7, 9-11 /home/ntfs-3g-pjd-fstest/tests/open/07.t (Wstat: 0 Tests: 23 Failed: 3) Failed tests: 5, 7, 9 Files=210, Tests=2328, 112 wallclock secs ( 0.89 usr 0.27 sys + 10.91 cusr 17.44 csys = 29.51 CPU) Result: FAIL Version-Release number of selected component (if applicable): # rpm -qa | grep ganesha nfs-ganesha-2.5.5-9.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.5.5-9.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-15.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-9.el7rhgs.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.Create 6*3 Distributed-Replicate Volume. 2.Export the volume via ganesha 3.Mount the volume on client via v4 protocol 4.Run posix compliance test suit Actual results: ftruncate/00.t tests failed in posix test suit for v4 and v3 .open/00.t tests failed for v4 mount Expected results: These test should have passed.This is a regression from earlier ganesha bits. Additional info: ganesha.log has following failures ------------- 10/08/2018 14:41:05 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-13488[work-205] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Permission denied 10/08/2018 14:41:05 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-13488[work-207] glusterfs_close_my_fd :FSAL :CRIT :Error : close returns with Permission denied ------------- ganesha-gfapi.log ------------ [2018-08-10 09:11:03.157674] E [dht-helper.c:90:dht_fd_ctx_set] (-->/usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x2e97c) [0x7f105a91397c] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x6f5d3) [0x7f105a6b05d3] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x7936) [0x7f105a648936] ) 0-mani2-dht: invalid argument: fd [Invalid argument] [2018-08-10 09:11:05.326586] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-mani2-client-1: remote operation failed. Path: /fstest_70d0247fdd803ed1fd55d05cc5f20b1c/fstest_4c9a97cc1f221471cd13d83d964f6025 (8a624d10-5f07-4a11-998b-c65bf1df7fbf) [Permission denied] [2018-08-10 09:11:05.326705] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-mani2-client-2: remote operation failed. Path: /fstest_70d0247fdd803ed1fd55d05cc5f20b1c/fstest_4c9a97cc1f221471cd13d83d964f6025 (8a624d10-5f07-4a11-998b-c65bf1df7fbf) [Permission denied] [2018-08-10 09:11:05.326769] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-mani2-client-0: remote operation failed. Path: /fstest_70d0247fdd803ed1fd55d05cc5f20b1c/fstest_4c9a97cc1f221471cd13d83d964f6025 (8a624d10-5f07-4a11-998b-c65bf1df7fbf) [Permission denied] [2018-08-10 09:11:45.962011] E [dht-helper.c:90:dht_fd_ctx_set] (-->/usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x2e97c) [0x7f105a91397c] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x6f5d3) [0x7f105a6b05d3] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x7936) [0x7f105a648936] ) 0-mani2-dht: invalid argument: fd [Invalid argument] ------------------------- Ran each of these case individually as well-- root@rhs-client6 ganesha]# prove -r /home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t /home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t .. Failed 1/26 subtests Test Summary Report ------------------- /home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t (Wstat: 0 Tests: 26 Failed: 1) Failed test: 24 Files=1, Tests=26, 2 wallclock secs ( 0.03 usr 0.01 sys + 0.09 cusr 0.17 csys = 0.30 CPU) Result: FAIL [root@rhs-client6 ganesha]# prove -r /home/ntfs-3g-pjd-fstest/tests/open/00.t /home/ntfs-3g-pjd-fstest/tests/open/00.t .. Failed 5/47 subtests Test Summary Report ------------------- /home/ntfs-3g-pjd-fstest/tests/open/00.t (Wstat: 0 Tests: 47 Failed: 5) Failed tests: 30-34 Files=1, Tests=47, 4 wallclock secs ( 0.04 usr 0.00 sys + 0.15 cusr 0.26 csys = 0.45 CPU) Result: FAIL
Marking this as blocker, its a regression. Ftruncate was fixed in earlier ganesha bits - https://bugzilla.redhat.com/show_bug.cgi?id=1452161
Manisha noted that she has seen the permission errors in other circumstances and noted that https://bugzilla.redhat.com/show_bug.cgi?id=1580120 was a recent permission related issue and we wonder if the fix for that could be a cause, so I have asked Manisha to test with the glusterfs-3.12.2-12 build to test without that fix. These tests run without error on FSAL_VFS on Ganesha next branch, and I don't recall any recent permission fixes, so I suspect the issue is FSAL_GLUSTER or gfapi related.
Setting needinfo on Manisha to come back with the results as requested in comment 3.
Changing the summary of the Bug.Keeping this BZ for ftruncate/00.t test failure. Will open a new BZ for open/00.t case once ran with debug logs enabled.
This failure occurs as the control never reaches to the ftruncate part but throws EPERM while fetching attributes(mtime...) for the file. The glusterfs_setattr2 func calls glfs_h_setattrs() from glusterfs which uses object-handle and not the file descriptor, hence fetching the attribute fails with EPERM. This is not needed to be fixed since fixing this would cause design compromise.