This bug is to document the steps needed to setup and run NFS-Ganesha as non-root user. Admin-guide link : https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/ Section: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/sect-nfs#sect-NFS_Ganesha We shall provide the detailed steps sooner.
Here is a first pass at the steps required 1) create ganesha user in /etc/passwd, e.g. I used UID 30. + ganesha:x:30:30:NFS-Ganesha Daemon:/etc/ganesha:/sbin/nologin and group in /etc/group + ganesha:x:30: Issue: need to pick UID and GID that doesn't collide with anything else. We would want a "Soft Static" allocation as per https://fedoraproject.org/wiki/Packaging:UsersAndGroups. 2) Add User=ganesha in /usr/lib/systemd/system/nfs-ganesha.service Service block and run `systemctl daemon-reload`. Issue: systemctl daemon-reload seemed unreliable when I tried it. A reboot might be better. 3) add a policy block for the ganesha user in /etc/dbus-1/system.d/org.ganesha.nfsd.conf and restart dbus daemon. 4) optional reboot in case the `systemctl daemon-reload` and/or restarting the dbus daemon are unreliable (which they have been seen to be.) 5) make /var/log/, /var/run, /var/run/ganesha, and /var/lib/nfs/ganesha/* world writeable Issue: subsequent reboots will reset perms on directories — at least /var/log and /var/run. We could move logs to /var/log/ganesha/ and use a /etc/tmpfiles.d/ganesha.conf for /var/log/ganesha, /var/run/ganesha, and /var/lib/nfs/ganesha). Issue: We could move ganesha.pid to somewhere else, e.g. /var/run/ganesha/ganesha.pid. The HA resource agents that expect /var/run/ganesha.pid would have to be changed accordingly. 6) /etc/ganesha/ganesha.conf changes: + Set Enable_RQUOTA = false; in NFS_Core_Param block. + Set Protocols = "4"; in each EXPORT block. + Set Transports = "TCP"; in each EXPORT block Issue: with RQUOTA enabled, ganesha tries to create a udp listener, and there is currently no way to disable only the udp listener. Issue: NFSv3 works, with log warnings about udp registration sometimes.
Second pass: 1. Create a ganesha user and group on each node where ganesha.nfsd runs % getent ganesha > /dev/null || groupadd -r ganesha % getent passwd ganesha > /dev/null || useradd -r -g ganesha -d /etc/ganesha -s /sbin/nologin -c "NFS-Ganesha Daemon" ganesha 2. Modify the systemd nfs-ganesha.service file to run the ganesha.nfsd as user ganesha by adding "User=ganesha" to the Service block. ... [Service] User=ganesha ... 3. Add a ganesha user to the dbus policy for ganesha.nfsd. Copy the policy user="root" block. In the second copy, change the user from "root" to "ganesha" ... <policy user="root"> <allow own="org.ganesha.nfsd"/> <allow send_destination="org.ganesha.nfsd"/> </policy> <policy user="ganesha"> <allow own="org.ganesha.nfsd"/> <allow send_destination="org.ganesha.nfsd"/> </policy> ... 4. Create /var/log/ganesha directory % mkdir /var/log/ganesha % chown ganesha:ganesha /var/log/ganesha % chmod 0775 /var/log/ganesha 5. Move ganesha's main log file from /var/log/ganesha.log to /var/log/ganesha/ganesha.log. Edit /etc/sysconfig/ganesha. ... OPTIONS="-L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT" ... 6. Create an /etc/tmpfiles.d/ganesha.conf file to mitigate the need for world-writable directories containing two lines as follows: D /var/run/ganesha 0755 ganesha ganesha - D /var/lib/nfs/ganesha 0755 ganesha ganesha - 7. Restart/reload systemd and dbus. (N.B. daemon-reload did not seem reliable when I tried it; work-around by rebooting the node(s).) % systemctl daemon-reload % systemctl restart dbus 8. make /var/log/, /var/run, /var/run/ganesha, and /var/lib/nfs/ganesha/* world writeable. (N.B. system will reset perms after a reboot.) % chmod 0777 /var/log /var/run /var/run/ganesha % find /var/lib/nfs/ganesha -type d -exec chmod 0777 {} \; 9. Modify /etc/ganesha/ganesha.conf to disable rquota (may not actually be necessary), adding an NFS_Core_Param block: ... NFS_Core_Param { Enable_RQUOTA = false; } ...
Second pass plus fix a typo: 1. Create a ganesha user and group on each node where ganesha.nfsd runs % getent group ganesha > /dev/null || groupadd -r ganesha % getent passwd ganesha > /dev/null || useradd -r -g ganesha -d /etc/ganesha -s /sbin/nologin -c "NFS-Ganesha Daemon" ganesha 2. Modify the systemd nfs-ganesha.service file to run the ganesha.nfsd as user ganesha by adding "User=ganesha" to the Service block. ... [Service] User=ganesha ... 3. Add a ganesha user to the dbus policy for ganesha.nfsd. Copy the policy user="root" block. In the second copy, change the user from "root" to "ganesha" ... <policy user="root"> <allow own="org.ganesha.nfsd"/> <allow send_destination="org.ganesha.nfsd"/> </policy> <policy user="ganesha"> <allow own="org.ganesha.nfsd"/> <allow send_destination="org.ganesha.nfsd"/> </policy> ... 4. Create /var/log/ganesha directory % mkdir /var/log/ganesha % chown ganesha:ganesha /var/log/ganesha % chmod 0775 /var/log/ganesha 5. Move ganesha's main log file from /var/log/ganesha.log to /var/log/ganesha/ganesha.log. Edit /etc/sysconfig/ganesha. ... OPTIONS="-L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT" ... 6. Create an /etc/tmpfiles.d/ganesha.conf file to mitigate the need for world-writable directories containing two lines as follows: D /var/run/ganesha 0755 ganesha ganesha - D /var/lib/nfs/ganesha 0755 ganesha ganesha - 7. Restart/reload systemd and dbus. (N.B. daemon-reload did not seem reliable when I tried it; work-around by rebooting the node(s).) % systemctl daemon-reload % systemctl restart dbus 8. make /var/log/, /var/run, /var/run/ganesha, and /var/lib/nfs/ganesha/* world writeable. (N.B. system will reset perms after a reboot.) % chmod 0777 /var/log /var/run /var/run/ganesha % find /var/lib/nfs/ganesha -type d -exec chmod 0777 {} \; 9. Modify /etc/ganesha/ganesha.conf to disable rquota (may not actually be necessary), adding an NFS_Core_Param block: ... NFS_Core_Param { Enable_RQUOTA = false; } ...
Hi kaleb, I created Ganesha setup with non-root user following the steps mentioned in comment #7 Ganesha setup came up and running with non-root user.Just an extra step is needed to be included. In step 5,where we are moving /var/log/ganesha.conf file to /var/log/ganesha/ganesha.conf,We need to provide permissions and change owner of ganesha.conf file as setting the permission to ganesha folder is not inheriting the permissions for the files inside ganesha folder. % chown ganesha:ganesha /var/log/ganesha/ganesha.conf % chmod 0775 /var/log/ganesha/ganesha.log. Observation: After creating ganesha setup with non-root user,I created a 3*2 Distributed-Replicate volume.Enabled ganesha on it. I mounted the volume via v3 on nfs client.Created files with touch command.So While creating files on mount point "Permission denied" messages were observed on mount point.But after doing "ls" on mount point ,the file were present on mount point. [root@dhcp37-192 ganesha_Mount]# touch f1 touch: setting times of ‘f1’: Permission denied [root@dhcp37-192 ganesha_Mount]# touch f2 touch: setting times of ‘f2’: Permission denied [root@dhcp37-192 ganesha_Mount]# ls f1 f2 In brick logs where file is present,these messages were observed while creating and opening file [2017-05-03 09:20:53.722045] I [MSGID: 115072] [server-rpc-fops.c:1642:server_setattr_cbk] 0-ganeshaVol1-server: 206: SETATTR /f1 (9aba96b8-8547-48e6-88db-ab0631ead6be) ==> (Permission denied) [Permission denied] [2017-05-03 09:21:42.221003] I [MSGID: 115072] [server-rpc-fops.c:1642:server_setattr_cbk] 0-ganeshaVol1-server: 224: SETATTR /.f1.swpx (9f47540f-2802-424c-ad0c-fda14bc7a118) ==> (Permission denied) [Permission denied] [2017-05-03 09:21:42.240186] E [MSGID: 115062] [server-rpc-fops.c:1037:server_unlink_cbk] 0-ganeshaVol1-server: 228: UNLINK /.f1.swpx (00000000-0000-0000-0000-000000000001/.f1.swpx) ==> (Permission denied) [Permission denied] [2017-05-03 09:21:47.266568] I [MSGID: 115072] [server-rpc-fops.c:1642:server_setattr_cbk] 0-ganeshaVol1-server: 247: SETATTR /.f1.swn (2ee2effc-cfc8-4fa9-8e9e-a628038d8eba) ==> (Permission denied) [Permission denied] [2017-05-03 09:21:47.286859] E [MSGID: 115062] [server-rpc-fops.c:1037:server_unlink_cbk] 0-ganeshaVol1-server: 251: UNLINK /.f1.swn (00000000-0000-0000-0000-000000000001/.f1.swn) ==> (Permission denied) [Permission denied] On bricks,the file are being created as root user [root@dhcp47-141 1]# ll total 0 -rw-r--r--. 2 root root 0 May 3 14:50 f1 -rw-r--r--. 2 root root 0 May 3 15:28 f2
(In reply to Manisha Saini from comment #8) > After creating ganesha setup with non-root user,I created a 3*2 > Distributed-Replicate volume.Enabled ganesha on it. > I mounted the volume via v3 on nfs client.Created files with touch > command.So While creating files on mount point "Permission denied" messages > were observed on mount point.But after doing "ls" on mount point ,the file > were present on mount point. Just to confirm, the customer is experiencing the exact same problem so it's not specific to this setup. At the customer, this "permission denied" does not occur always so it's intermittent and so far they where not able to resolve it.
Just to document further, /var/log/ganesha/ganesha.log has 05/05/2017 18:55:05 : epoch 08600000 : dhcp47-141.lab.eng.blr.redhat.com : ganesha.nfsd-31394[work-89] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Permission denied and /var/log/ganesha-gfapi.log has [2017-05-05 13:25:05.126379] W [MSGID: 114031] [client-rpc-fops.c:2266:client3_3_setattr_cbk] 0-ganeshaVol1-client-3: remote operation failed [Permission denied] Which is all consistent with the other logs above.
Created attachment 1277160 [details] network capture with NFS+GlusterFS packets for failing touch
Created attachment 1277161 [details] text file with the interesting packets generated with tshark
Created attachment 1277162 [details] annotated packets with explanations and conclusion
All details are in attachment 1277162 [details] which explains why the 'permission denied' errors are seen with 'touch'. This points to a bug in NFS-Ganesha (most probably). > # Conclusion > > NFS-Ganesha most likely fails to call glfs_setfsuid(), glfs_setfsgid() and > glfs_setfsgroups() in some circumstances. Most likely the UID/GID that is > seen (48/48) is the UID/GID of the NFS-Gansesh non-root user.
Hmm, one thing I would consider, if one is running Ganesha as non-root, it is appropriate to expect it to be able to perform actions on behalf of root? What does glfs_setfsuid actually do? Depending on what it actually does, either non-root Ganesha needs additional capabilities, or libglusterfs needs to allow non-root Ganesha to assume root identity for operations on behalf of root.
(In reply to Frank Filz from comment #15) > Hmm, one thing I would consider, if one is running Ganesha as non-root, it > is appropriate to expect it to be able to perform actions on behalf of root? > > What does glfs_setfsuid actually do? Depending on what it actually does, > either non-root Ganesha needs additional capabilities, or libglusterfs needs > to allow non-root Ganesha to assume root identity for operations on behalf > of root. glfs_setfsuid() allows the applications to set the thread ctx uid/gid to the ones passed as arguments. These credentials are used while sending any operations to glusterfs brick processes. There is no capability in glusterfs to allow any non-root user to have root access [I am not sure if any other filesystem has this support. Tried checking FSAL_VFS - but mount point itself was not accessible when the nfs-ganesha service is started by non-root user] Probably we may need fix in ganesha sources itself to pass opctx->uid/gid all the time (but not just for create/write operations) before doing any backend operations.
(In reply to Frank Filz from comment #15) > Hmm, one thing I would consider, if one is running Ganesha as non-root, it > is appropriate to expect it to be able to perform actions on behalf of root? Yes, the credentials passed from NFS-Ganesha to Gluster are similar to AUTH_UNIX in the NFS world; it is just a UID/GID/EGID set. > What does glfs_setfsuid actually do? Depending on what it actually does, > either non-root Ganesha needs additional capabilities, or libglusterfs needs > to allow non-root Ganesha to assume root identity for operations on behalf > of root. glfs_setfsuid() and friends set the UID/GID/EGID that are used for credentials in the GlusterFS protocol. As Soumya mentioned, these need to be set in the thread that does the calling of glfs_[f]setattr(). NFS-Ganesha seems to call glfs_creat() with the correct credentials set, this should be done for other procedures as well.
Ok, I think the issue is just that we don't call glfs_setfsuid etc. on set attributes. It looks like FSAL_VFS doesn't either... But there should be no downside to setting proper credentials for set attributes, or any other operation that changes the filesystem. There is on downside to setting credentials for get attributes. In a system where there are NFS v4 ACLs or Rich ACLs underneath, a READ_ACL could be denied which would prevent Ganesha from getting the ACL so that it can do permission checks (READ_ATTR would also be an issue there, though most likely any filesystem intended for use in a POSIX environment would not enforce READ_ATTR - though that could come up in the an SE Linux situation).
@manisha, Could you please file a RHGS bug for the issue you mentioned in comment#8 and make this bug dependent on that one. We need to address that prior to documenting these steps.
(In reply to Soumya Koduri from comment #19) > @manisha, > > Could you please file a RHGS bug for the issue you mentioned in comment#8 > and make this bug dependent on that one. We need to address that prior to > documenting these steps. Soumya, I have created a separate bug to track the issue reported in comment #8 https://bugzilla.redhat.com/show_bug.cgi?id=1450836
Correcting the step I mentioned in comment #8 It should be ganesha.log instead of ganesha.conf https://docs.google.com/a/redhat.com/document/d/1pbYiPvoRfk2XDv7gKK1oLsu5kEKyJwFubSa8-dbwf1w/edit?usp=sharing 6. Provide the permissions to the ganesha user to write to the log file: % chown ganesha:ganesha /var/log/ganesha/ganesha.log % chmod 0775 /var/log/ganesha/ganesha.log.
This bug needs to be closed as we are not targeting non-root user requirement to document in 3.3. Hence closing this bug as NOTABUG based on comment #32
Since this bug is been targeted for future release,mistakenly I moved it to CLOSED NOTABUG. The engineering bug has been CLOSED but since its a documentation bug,this can be moved to future release. Moving the bug state again to ON-QA. Requesting anjana to remove the 3.3.0 keyword and reset the flags according to future release
As the engineering bug "https://bugzilla.redhat.com/show_bug.cgi?id=1439117" has moved to 3.5.0 , so we have changed this document bug internal whiteboard to 3.5.0.
*** Bug 1695079 has been marked as a duplicate of this bug. ***
Setting needinfo on DEV to review as well
Changes looks good to me.Moving this BZ to verified state
5+ years old, RHGS is EOL, was VERIFIED and never closed. closing