Bug 1380742 - Some tests in pynfs test suite fails with latest 3.2 builds.
Summary: Some tests in pynfs test suite fails with latest 3.2 builds.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: RHGS 3.2.0
Assignee: Soumya Koduri
QA Contact: Arthy Loganathan
URL:
Whiteboard:
Depends On: 1381968
Blocks: 1351528
TreeView+ depends on / blocked
 
Reported: 2016-09-30 12:58 UTC by Shashank Raj
Modified: 2017-03-23 06:07 UTC (History)
9 users (show)

Fixed In Version: glusterfs-3.8.4-4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-23 06:07:21 UTC


Attachments (Terms of Use)
OPEN2_without_mdcache (4.94 MB, application/octet-stream)
2016-10-05 12:46 UTC, Soumya Koduri
no flags Details
OPEN2_with_mdcache (7.44 MB, application/octet-stream)
2016-10-05 12:47 UTC, Soumya Koduri
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Shashank Raj 2016-09-30 12:58:13 UTC
Description of problem:

Some tests in pynfs test suite fails with latest 3.2 builds.

Version-Release number of selected component (if applicable):

[root@dhcp43-110 ~]# rpm -qa|grep glusterfs
glusterfs-api-3.8.4-1.el7rhgs.x86_64
glusterfs-fuse-3.8.4-1.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-1.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-1.el7rhgs.x86_64
glusterfs-libs-3.8.4-1.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-1.el7rhgs.x86_64
glusterfs-cli-3.8.4-1.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-1.el7rhgs.x86_64
glusterfs-3.8.4-1.el7rhgs.x86_64
glusterfs-server-3.8.4-1.el7rhgs.x86_64
glusterfs-events-3.8.4-1.el7rhgs.x86_64

[root@dhcp43-110 ~]# rpm -qa|grep ganesha
nfs-ganesha-debuginfo-2.4.0-2.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.0-2.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-1.el7rhgs.x86_64
nfs-ganesha-2.4.0-2.el7rhgs.x86_64


How reproducible:

Always

Steps to Reproduce:
1. Create a nfs-ganesha cluster.
2. Create a volume, enable ganesha and mount the volume with vers 4.
3. Run pynfs test suite from the client

./testserver.py 10.70.40.209:/testvolume -v --outfile ~/pynfs.run --maketree --showomit --rundep all > /tmp/pynfs-32-build.log

4. Observe that some of the tests related to OPEN2 and WRT18 fails.

OPEN2    st_open.testCreateUncheckedFile                          : FAILURE
           Attrs on recreate should be ignored, except for size:
           For size expected 0, got 32

WRT18    st_write.testChangeGranularityWrite                      : FAILURE
           consecutive SETATTR(mode)'s don't all change change
           attribute

OPEN2    st_open.testCreateUncheckedFile                          : FAILURE
           Attrs on recreate should be ignored, except for size:
           For size expected 0, got 32

WRT18    st_write.testChangeGranularityWrite                      : FAILURE
           consecutive SETATTR(mode)'s don't all change change
           attribute


Actual results:

Some tests in pynfs test suite fails with latest 3.2 builds.

Expected results:

There should not be any failures

Additional info:

tcpdump will be attached.

Comment 2 Shashank Raj 2016-09-30 13:00:49 UTC
tcpdumps are at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1380742

Comment 3 Soumya Koduri 2016-10-05 12:46:17 UTC
Created attachment 1207588 [details]
OPEN2_without_mdcache

Comment 4 Soumya Koduri 2016-10-05 12:47:20 UTC
Created attachment 1207589 [details]
OPEN2_with_mdcache

Comment 5 Soumya Koduri 2016-10-05 12:47:49 UTC
WRT18 (compound WRITEs) is currently does not seem to be supported by most of existing NFS server implementations including kernel-NFS. And it is not a generic usecase as well.

wrt OPEN2, there is OPEN on a file followed by GETATTR which expects size to be '0'.  But in case gluster, this GETATTR was served by md-cache which returned old stat of the file. If we disable md-cache (stat-prefetch to off), we can see this testcase pass. Attached the pkt traces taken which confirms this behavior. There is not FSTAT call sent to Glusterfs server when md-cache is enabled.

We need to fix md-cache to handle open() fop as well and invalidate its cache if the open flags contain O_TRUNC. Will open another bug for it.

Comment 6 Soumya Koduri 2016-10-05 12:53:21 UTC
Filed bug1381968 to handle the issue with md-cache.

Comment 7 Niels de Vos 2016-10-05 13:57:39 UTC
The pynfs command that we execute for upstream NFS-Ganesha testing is located here:

  https://github.com/nfs-ganesha/ci-tests/blob/centos-ci/pynfs/client.sh#L27

The job translates that to this:

  ./testserver.py n27.dusty.ci.centos.org:/pynfs --verbose --maketree --showomit --rundeps all ganesha noWRT18 noRD4 noSEC7


Frank suggested these options, it enables the tests that NFS-Ganesha is expected to pass (when Kerberos is not configured). Could you check if you run the latest version of pynfs and the test passes when executed like upstream?

Comment 8 Shashank Raj 2016-10-06 06:59:20 UTC
Niels,

After running pynfs with below:

 ./testserver.py 10.70.40.209:/testvolume --verbose --maketree --showomit --rundeps all ganesha noWRT18 noRD4 noSEC7

i see OPEN2 related failure, which is already mentioned in the description of the bug

OPEN2    st_open.testCreateUncheckedFile                          : FAILURE
           Attrs on recreate should be ignored, except for size:
           For size expected 0, got 32

Comment 9 Soumya Koduri 2016-10-06 07:49:54 UTC
Patch for bug1381968 is posted upstream for review.

Comment 12 Atin Mukherjee 2016-11-07 08:39:23 UTC
upstream mainline: http://review.gluster.org/15618 
upstream 3.8: http://review.gluster.org/15771
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/88936/

Comment 16 Arthy Loganathan 2016-11-18 08:39:58 UTC
Verified the fix in build,

nfs-ganesha-2.4.1-1.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64

Comment 18 errata-xmlrpc 2017-03-23 06:07:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.