Description of problem: Some tests in pynfs test suite fails with latest 3.2 builds. Version-Release number of selected component (if applicable): [root@dhcp43-110 ~]# rpm -qa|grep glusterfs glusterfs-api-3.8.4-1.el7rhgs.x86_64 glusterfs-fuse-3.8.4-1.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-1.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-1.el7rhgs.x86_64 glusterfs-libs-3.8.4-1.el7rhgs.x86_64 glusterfs-client-xlators-3.8.4-1.el7rhgs.x86_64 glusterfs-cli-3.8.4-1.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-1.el7rhgs.x86_64 glusterfs-3.8.4-1.el7rhgs.x86_64 glusterfs-server-3.8.4-1.el7rhgs.x86_64 glusterfs-events-3.8.4-1.el7rhgs.x86_64 [root@dhcp43-110 ~]# rpm -qa|grep ganesha nfs-ganesha-debuginfo-2.4.0-2.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.0-2.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-1.el7rhgs.x86_64 nfs-ganesha-2.4.0-2.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1. Create a nfs-ganesha cluster. 2. Create a volume, enable ganesha and mount the volume with vers 4. 3. Run pynfs test suite from the client ./testserver.py 10.70.40.209:/testvolume -v --outfile ~/pynfs.run --maketree --showomit --rundep all > /tmp/pynfs-32-build.log 4. Observe that some of the tests related to OPEN2 and WRT18 fails. OPEN2 st_open.testCreateUncheckedFile : FAILURE Attrs on recreate should be ignored, except for size: For size expected 0, got 32 WRT18 st_write.testChangeGranularityWrite : FAILURE consecutive SETATTR(mode)'s don't all change change attribute OPEN2 st_open.testCreateUncheckedFile : FAILURE Attrs on recreate should be ignored, except for size: For size expected 0, got 32 WRT18 st_write.testChangeGranularityWrite : FAILURE consecutive SETATTR(mode)'s don't all change change attribute Actual results: Some tests in pynfs test suite fails with latest 3.2 builds. Expected results: There should not be any failures Additional info: tcpdump will be attached.
tcpdumps are at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1380742
Created attachment 1207588 [details] OPEN2_without_mdcache
Created attachment 1207589 [details] OPEN2_with_mdcache
WRT18 (compound WRITEs) is currently does not seem to be supported by most of existing NFS server implementations including kernel-NFS. And it is not a generic usecase as well. wrt OPEN2, there is OPEN on a file followed by GETATTR which expects size to be '0'. But in case gluster, this GETATTR was served by md-cache which returned old stat of the file. If we disable md-cache (stat-prefetch to off), we can see this testcase pass. Attached the pkt traces taken which confirms this behavior. There is not FSTAT call sent to Glusterfs server when md-cache is enabled. We need to fix md-cache to handle open() fop as well and invalidate its cache if the open flags contain O_TRUNC. Will open another bug for it.
Filed bug1381968 to handle the issue with md-cache.
The pynfs command that we execute for upstream NFS-Ganesha testing is located here: https://github.com/nfs-ganesha/ci-tests/blob/centos-ci/pynfs/client.sh#L27 The job translates that to this: ./testserver.py n27.dusty.ci.centos.org:/pynfs --verbose --maketree --showomit --rundeps all ganesha noWRT18 noRD4 noSEC7 Frank suggested these options, it enables the tests that NFS-Ganesha is expected to pass (when Kerberos is not configured). Could you check if you run the latest version of pynfs and the test passes when executed like upstream?
Niels, After running pynfs with below: ./testserver.py 10.70.40.209:/testvolume --verbose --maketree --showomit --rundeps all ganesha noWRT18 noRD4 noSEC7 i see OPEN2 related failure, which is already mentioned in the description of the bug OPEN2 st_open.testCreateUncheckedFile : FAILURE Attrs on recreate should be ignored, except for size: For size expected 0, got 32
Patch for bug1381968 is posted upstream for review.
upstream mainline: http://review.gluster.org/15618 upstream 3.8: http://review.gluster.org/15771 downstream patch : https://code.engineering.redhat.com/gerrit/#/c/88936/
Verified the fix in build, nfs-ganesha-2.4.1-1.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html