Bug 1439117

Description of problem:
Need to run Ganesha NFS as non-root user, due to company internal security policies. 

Version-Release number of selected component (if applicable):
RHGS 3.2

How reproducible:
Run NFS-Ganesha processes as a non-root user, see addl info.

Steps to Reproduce:
1.Run NFS-Ganesha as non-root
2.Write a file, success
3.Write another file -> zero bytes error (happens randomly)

Actual results:
Sometimes normal file write, sometimes zero bytes file error

Expected results:
Normal file write

Additional info:

What is tried to accomplish is having Ganesha running as a non-root user (in this case, opted for the "hacluster" user). 
Outside the experienced write errors, all other items seem to be working quite fine. All Ganesha related processes report to be running fine, all pacemaker/corosync resources are up/running (failover works fine as well).

Modifications to have Ganesha running withour root rights:
Ganesha system service file to include a: user=hacluster
The pid file and logfile location in an alternate directory, and therefor also modify a pacemaker probe script which had a hardcoded location for the ganesha.pid file.

The RQUOTA port defaults to 875, and therefore needed be > 1024  (set to 1875).

Inside /etc/dbus-1/system.d/org.ganesha.nfsd.conf, the policy user needed to be updates with hacluster instead of root.
Several directories needed to be chowned to match the hacluster user such as:
/var/run/gluster/shared_storage/nfs-ganesha  (recursive)
<brick_path>/<ganesha volume mountpoint>

Apart from the Ganesha logs and brick logs, no other error are seen.

This is RHGS 3.2 on RHEL 7.3.
SELinux is on (Enforcing), (no issues with SElinux spotted)

NFS Ganesha running as non-root and
clients experiences write errors / permission denied. 

Files written to a NFS-ganesha share, mounted on a client, get a write error/permission denied. Sometimes no error is returned, but no data is actually written (a zero byte file)

Sometimes, writes do actually complete correctly, sometimes they do not...

A correctly written file :
[glustadm@tlrvglusterclient02 XFB_LV]$ id > FILE_2 ; ls -l FILE_2
-rw-rw-r--. 1 glustadm glustadm 125 Mar 23 17:17 FILE_2

1 second later, a failure:
[glustadm@tlrvglusterclient02 XFB_LV]$ id > FILE_3 ; ls -l FILE_3
id: write error: Permission denied
-rw-rw-r--. 1 glustadm glustadm 0 Mar 23 17:17 FILE_3

Repeating it (to the same file as previously failed), gets first a few additional failures, then completes:
[glustadm@tlrvglusterclient02 XFB_LV]$ id > FILE_3 ; ls -l FILE_3
-rw-rw-r--. 1 glustadm glustadm 0 Mar 23 17:17 FILE_3

[glustadm@tlrvglusterclient02 XFB_LV]$ id > FILE_3 ; ls -l FILE_3
-rw-rw-r--. 1 glustadm glustadm 125 Mar 23 17:20 FILE_3