Bug 1614712

Summary: [Ganesha] ftruncate/00.t is failing in posix compliance test suit for ganesha
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: nfs-ganeshaAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED WONTFIX QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: amukherj, arjsharm, dang, ffilz, grajoria, jthottan, mbenjamin, msaini, pasik, rhs-bugs, sankarshan, sheggodu, skoduri, storage-qa-internal
Target Milestone: ---Keywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-29 12:16:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2018-08-10 09:29:38 UTC 
Description of problem:

while running posix compliance test suit on ganesha mount, ftruncate/00.t and open/00.t tests are failing.
-----
3.4
-----

Test Summary Report
-------------------
/home/ntfs-3g-pjd-fstest/tests/chmod/00.t    (Wstat: 0 Tests: 106 Failed: 2)
  Failed tests:  31, 39
/home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t (Wstat: 0 Tests: 26 Failed: 1)
  Failed test:  24
/home/ntfs-3g-pjd-fstest/tests/mknod/03.t    (Wstat: 0 Tests: 12 Failed: 9)
  Failed tests:  1-3, 5-7, 9-11
/home/ntfs-3g-pjd-fstest/tests/open/00.t     (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
/home/ntfs-3g-pjd-fstest/tests/open/07.t     (Wstat: 0 Tests: 23 Failed: 3)
  Failed tests:  5, 7, 9
Files=210, Tests=2328, 92 wallclock secs ( 0.84 usr  0.29 sys + 10.93 cusr 16.82 csys = 28.88 CPU)
Result: FAIL

-----
3.3.1
-----
Test Summary Report
-------------------
/home/ntfs-3g-pjd-fstest/tests/chmod/00.t    (Wstat: 0 Tests: 106 Failed: 2)
  Failed tests:  31, 39
/home/ntfs-3g-pjd-fstest/tests/mknod/03.t    (Wstat: 0 Tests: 12 Failed: 9)
  Failed tests:  1-3, 5-7, 9-11
/home/ntfs-3g-pjd-fstest/tests/open/07.t     (Wstat: 0 Tests: 23 Failed: 3)
  Failed tests:  5, 7, 9
Files=210, Tests=2328, 112 wallclock secs ( 0.89 usr  0.27 sys + 10.91 cusr 17.44 csys = 29.51 CPU)
Result: FAIL



Version-Release number of selected component (if applicable):

# rpm -qa | grep ganesha
nfs-ganesha-2.5.5-9.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.5.5-9.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-15.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-9.el7rhgs.x86_64


How reproducible:
3/3

Steps to Reproduce:
1.Create 6*3 Distributed-Replicate Volume.
2.Export the volume via ganesha
3.Mount the volume on client via v4 protocol
4.Run posix compliance test suit

Actual results:

ftruncate/00.t tests failed in posix test suit for v4 and  v3 .open/00.t tests failed for v4 mount

Expected results:

These test should have passed.This is a regression from earlier ganesha bits.

Additional info:

ganesha.log has following failures

-------------
10/08/2018 14:41:05 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-13488[work-205] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Permission denied
10/08/2018 14:41:05 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-13488[work-207] glusterfs_close_my_fd :FSAL :CRIT :Error : close returns with Permission denied
-------------

ganesha-gfapi.log

------------
[2018-08-10 09:11:03.157674] E [dht-helper.c:90:dht_fd_ctx_set] (-->/usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x2e97c) [0x7f105a91397c] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x6f5d3) [0x7f105a6b05d3] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x7936) [0x7f105a648936] ) 0-mani2-dht: invalid argument: fd [Invalid argument]
[2018-08-10 09:11:05.326586] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-mani2-client-1: remote operation failed. Path: /fstest_70d0247fdd803ed1fd55d05cc5f20b1c/fstest_4c9a97cc1f221471cd13d83d964f6025 (8a624d10-5f07-4a11-998b-c65bf1df7fbf) [Permission denied]
[2018-08-10 09:11:05.326705] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-mani2-client-2: remote operation failed. Path: /fstest_70d0247fdd803ed1fd55d05cc5f20b1c/fstest_4c9a97cc1f221471cd13d83d964f6025 (8a624d10-5f07-4a11-998b-c65bf1df7fbf) [Permission denied]
[2018-08-10 09:11:05.326769] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-mani2-client-0: remote operation failed. Path: /fstest_70d0247fdd803ed1fd55d05cc5f20b1c/fstest_4c9a97cc1f221471cd13d83d964f6025 (8a624d10-5f07-4a11-998b-c65bf1df7fbf) [Permission denied]
[2018-08-10 09:11:45.962011] E [dht-helper.c:90:dht_fd_ctx_set] (-->/usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x2e97c) [0x7f105a91397c] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x6f5d3) [0x7f105a6b05d3] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x7936) [0x7f105a648936] ) 0-mani2-dht: invalid argument: fd [Invalid argument]

-------------------------


Ran each of these case individually as well--


root@rhs-client6 ganesha]# prove -r /home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t 
/home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t .. Failed 1/26 subtests 

Test Summary Report
-------------------
/home/ntfs-3g-pjd-fstest/tests/ftruncate/00.t (Wstat: 0 Tests: 26 Failed: 1)
  Failed test:  24
Files=1, Tests=26,  2 wallclock secs ( 0.03 usr  0.01 sys +  0.09 cusr  0.17 csys =  0.30 CPU)
Result: FAIL


[root@rhs-client6 ganesha]# prove -r /home/ntfs-3g-pjd-fstest/tests/open/00.t 
/home/ntfs-3g-pjd-fstest/tests/open/00.t .. Failed 5/47 subtests 

Test Summary Report
-------------------
/home/ntfs-3g-pjd-fstest/tests/open/00.t (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
Files=1, Tests=47,  4 wallclock secs ( 0.04 usr  0.00 sys +  0.15 cusr  0.26 csys =  0.45 CPU)
Result: FAIL
Comment 2 Manisha Saini 2018-08-10 09:32:38 UTC 
Marking this as blocker, its a regression.

Ftruncate was fixed in earlier ganesha bits -
https://bugzilla.redhat.com/show_bug.cgi?id=1452161
Comment 3 Frank Filz 2018-08-10 18:45:11 UTC 
Manisha noted that she has seen the permission errors in other circumstances and noted that https://bugzilla.redhat.com/show_bug.cgi?id=1580120 was a recent permission related issue and we wonder if the fix for that could be a cause, so I have asked Manisha to test with the glusterfs-3.12.2-12 build to test without that fix.

These tests run without error on FSAL_VFS on Ganesha next branch, and I don't recall any recent permission fixes, so I suspect the issue is FSAL_GLUSTER or gfapi related.
Comment 4 Atin Mukherjee 2018-08-14 07:05:54 UTC 
Setting needinfo on Manisha to come back with the results as requested in comment 3.
Comment 7 Manisha Saini 2018-08-16 09:18:25 UTC 
Changing the summary of the Bug.Keeping this BZ for ftruncate/00.t test failure.
Will open a new BZ for open/00.t case once ran with debug logs enabled.
Comment 12 Arjun 2019-04-29 12:16:59 UTC 
This failure occurs as the control never reaches to the ftruncate part but throws EPERM while fetching attributes(mtime...) for the file. The glusterfs_setattr2 func calls glfs_h_setattrs() from glusterfs which uses object-handle and not the file descriptor, hence fetching the attribute fails with EPERM.

This is not needed to be fixed since fixing this would cause design compromise.