Bug 1414663

Summary: [GANESHA] Cthon lock test case is failing on nfs-ganesha mounted Via V3
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: common-haAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: unspecified Docs Contact:
Priority: high    
Version: rhgs-3.2CC: amukherj, rcyriac, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-14 Doc Type: Bug Fix
Doc Text:
The rpc.statd process runs as the rpcuser user by default. Previously, the /var/lib/nfs/statd directory was not owned by the rpcuser user or group. This meant that when the rpc.statd process started before NFS-Ganesha had started, the rpc.statd process was unable to read or write client state to the /var/lib/nfs/statd directory. This directory and its files are now created with rpcuser as the owning user and group so that the rpc.statd process can access and maintain client state.
 Story Points: ---
Clone Of:
: 1414665 (view as bug list) Environment:
Last Closed: 2017-03-23 06:03:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1414665    
Bug Blocks: 1351528, 1351530    

Description Manisha Saini 2017-01-19 07:52:27 UTC 
Description of problem:
Cthon Lock test is failing on NFS-Ganesha V3

Version-Release number of selected component (if applicable):
glusterfs-ganesha-3.8.4-12.el6rhs.x86_64

How reproducible:
Consistently

Steps to Reproduce:
1.Setup 4 node ganesha cluster on Rhel6
2.Create 4*2 Distributed-Replicate Volume.Enable ganesha on it.
3.Export the Volume to Client via V3 on Rhel7 Client
4.Run cthon lock test suite
# cd /root/cthon04 && ./server -l -o vers=3 -p /replicaVol -m /mnt/rhel6/mountpoint/ -N 1 dhcp37-106.lab.eng.blr.redhat.com
sh ./runtests  -l -t /mnt/rhel6/mountpoint//dhcp37-192.test

Starting LOCKING tests: test directory /mnt/rhel6/mountpoint//dhcp37-192.test (arg: -t)

Testing native post-LFS locking

Creating parent/child synchronization pipes.

Test #1 - Test regions of an unlocked file.
	Parent: 1.1  - F_TEST  [               0,               1] PASSED.
	Parent: 1.2  - F_TEST  [               0,          ENDING] PASSED.
	Parent: 1.3  - F_TEST  [               0,7fffffffffffffff] PASSED.
	Parent: 1.4  - F_TEST  [               1,               1] PASSED.
	Parent: 1.5  - F_TEST  [               1,          ENDING] PASSED.
	Parent: 1.6  - F_TEST  [               1,7fffffffffffffff] PASSED.
	Parent: 1.7  - F_TEST  [7fffffffffffffff,               1] PASSED.
	Parent: 1.8  - F_TEST  [7fffffffffffffff,          ENDING] PASSED.
	Parent: 1.9  - F_TEST  [7fffffffffffffff,7fffffffffffffff] PASSED.

Test #2 - Try to lock the whole file.
	Parent: 2.0  - F_TLOCK [               0,          ENDING] FAILED!
	Parent: **** Expected success, returned errno=37...
	Parent: **** Probably implementation error.

** PARENT pass 1 results: 9/9 pass, 0/0 warn, 1/1 fail (pass/total).

**  CHILD pass 1 results: 0/0 pass, 0/0 warn, 0/0 fail (pass/total).
lock tests failed
Tests failed, leaving /mnt/rhel6/mountpoint/ mounted


Actual results:
Cthon lock test suit is failing with V3 and Passing with V4

Expected results:
Cthon Lock test suit should pass with V3

Additional info:
Comment 2 Soumya Koduri 2017-01-19 07:57:35 UTC 
Jan 18 19:35:17 dhcp47-74 rpc.statd[11770]: Failed to insert: creating /var/lib/nfs/statd/sm/dhcp37-192.lab.eng.blr.redhat.com: Permission denied
rpc.statd[11770]: STAT_FAIL to dhcp47-74.lab.eng.blr.redhat.com for SM_MON of dhcp37-192.lab.eng.blr.redhat.com

It seems to be Day-1 issue. statd process gets started as rpcuser. In the shared_storage nfs-ganesha folder, we need to chown the directories/files used by statd to 'rpcuser'.  Will send in a fix upstream for review.

The work-around is to either manually chown those files before setting up nfs-ganesha or restart statd process.
Comment 3 Soumya Koduri 2017-01-19 09:44:10 UTC 
patch posted usptream for review - http://review.gluster.org/16433
Comment 6 Manisha Saini 2017-02-07 14:26:28 UTC 
Verified this bug on glusterfs-3.8.4-14.el7rhgs.x86_64

As Cthon lock test case is passing successfully with nfs-ganesha v3 mount.Hence marking this bug as verified
Comment 8 Soumya Koduri 2017-03-08 06:55:25 UTC 
Hi Laura,

Doc text looks good.. but probably there was typo at the beginning. Please find the corrections below -

NLM Clients state could not be stored if the rpc.statd process started running prior to the NFS-Ganesha High Availability setup. The directory and its files where the state is maintained are now created with rpcuser as the owning user and group so that the rpc.statd process can access and maintain client state.
Comment 10 Soumya Koduri 2017-03-08 10:19:05 UTC 
The updates look good to me. Thanks!
Comment 12 errata-xmlrpc 2017-03-23 06:03:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html