Bug 1751210

Summary: [GSS][Permission denied errors observed when running 'git clone' command on a home directory exported by NFS-Ganesha]
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: nravinas
Component: nfs-ganeshaAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.4CC: amukherj, bkunal, dang, grajoria, jthottan, kkeithle, lbailey, mbenjamin, msaini, pasik, puebele, rhs-bugs, skoduri, storage-qa-internal, vdas
Target Milestone: ---   
Target Release: RHGS 3.5.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: nfs-ganesha-2.7.3-9 Doc Type: Bug Fix
Doc Text:
NFS-Ganesha used client credentials for all operations on Gluster storage. In cases where a non-root user was operating on a read-only file, this resulted in 'permission denied' errors. Root permissions are now used where appropriate so that non-root users are able to create and write to files using 0444 mode.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-30 12:15:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1735480, 1753569    
Bug Blocks: 1696810    
Attachments:
Description Flags
tcpdumps from the NFS-Ganesha server. none

Description nravinas 2019-09-11 12:31:16 UTC 
Created attachment 1614082 [details]
tcpdumps from the NFS-Ganesha server.

Description of problem:


* Running a 'git clone' command from a home directory mounted on an NFS-Ganesha exported volume shows 'Permission denied':

[user1@tendrl-0 ~]$ git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
fatal: write error: Permission denied0 MiB | 107.00 KiB/s

* Doing an strace, indeed this 'permission denied' issue comes at the time of writing to some tmp directory created inside the .git 
  folder:

[...]

write(4</home/user1/pvws/.git/objects/pack/tmp_pack_GDq5hd>, "\\P\34e\301\343XD\324\252z\302i\362C\251\252\17c\232{?\213\2016\212\252m\3516\224~\365Ye\236\274\2\365\236\336\4\234__\256d%4\3355\301k\375\356\216\35\261!c\337~\213\252\351\216}K\377\355\262\275~\237\35\36\262a\313!'\321\255\244V\37\207\327%\304\1\322:\r\366\332\303\310\332\340V\242\351)_l5\206D&G\227\5\307\357\340f\225\210]\274\213\334\344\34Y\354C\312Q\344\277\201\227\2001y\"}\34\35\266\347\273/\332\323\333\3\"Fq[G?\305\357\232\241\354n\315\236\361\363\255H-@L\363\364\211\n\26634t47\326K?C\271\241\375\320\230\320U\1\257\216\221\364\2640\177\373\324\21N\303\240#\232\26\263+\363*\314\0\215l\177O\201\342\260Z\235\1\26\250\rW\"\f\351\23 \265\257\244\35\215V\277\365f\32\364\357$\340k^o h\330\341i\361J\261\322\7eJ\224\v\325:\362\26\211\244Q\213O\231\270\230\273~\36q1_\207h\214\257L(?\254\16m]T\265\1M\21u\212-~\325\342\276\35\217N\320\16M'\377\34\327\255R\373E\213+\"*o\364\350\331aC\207\355\3569:\255:4{\244\244\200\374\21\313H\211\242+u\252\273\267\222E)\327u\30\304\275\306\363\rH\334\246\241\27\263\"5\233X\342i\207\26055\364\233\205A\v^\225\224w\177UX\366\31\215\27r\35AGU\347\301\233~\217\361'@\377\7\200\226\\\225\370\271\10\262\305\317\r\334\342\335\250\215\257\375/\3067X8\315.\312i7\\\277/\366-<\254\300\354/&\4\377\254\353\345FO\370\33x\6\r\237\20w\37P4}\300J\317\310\235\205~hBu\315\207\223\367\33\340_\354\271\301\372\235\231x\343\374z\323hm\23\344"..., 4096) = -1 EACCES (Permission denied) <5.090617>

[...]

* I've captured some tcpdumps at the time this problem was observed and there are 'NFS4ERR_ACCESS' errors clearly showing. They are attached to the BZ for your reference.
Comment 22 Manisha Saini 2019-10-09 09:43:14 UTC 
Verified this BZ with


# rpm -qa | grep ganesha
nfs-ganesha-2.7.3-9.el7rhgs.x86_64
glusterfs-ganesha-6.0-15.el7rhgs.x86_64
nfs-ganesha-gluster-2.7.3-9.el7rhgs.x86_64


Reproduced this issue with build nfs-ganesha-2.7.3-8.el7rhgs.x86_64

Steps:
1. Create 4 node ganesha cluster
2. Create a volume and export the volume via ganesha
3. Mount the volume on client via v 4.0
4. Provide permission 777 to ganesha mount dir
5. Create a user "mani"
6. Su mani
7. Perform  git clone https://github.com/kasemir/pvws.git on mount point


git clone failed with mount v 4.0,v4.1 but passed with v 3.0
v 4.0
-----------
bash-4.2$ git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
remote: Total 744 (delta 299), reused 669 (delta 224), pack-reused 0
Receiving objects: 100% (744/744), 5.00 MiB | 296.00 KiB/s, done.
Resolving deltas: 100% (299/299), done.
fatal: fsync error on '/home/pvws/.git/objects/pack/tmp_pack_gw83C2': Permission denied
fatal: index-pack failed
-------------

v 4.1
------------
bash-4.2$  git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
fatal: Unable to create temporary file '/home/pvws/.git/objects/pack/tmp_pack_XXXXXX': Permission denied
fatal: index-pack failed
---------------

v 3.0
------------
bash-4.2$  git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
remote: Total 744 (delta 299), reused 669 (delta 224), pack-reused 0
Receiving objects: 100% (744/744), 5.00 MiB | 1.18 MiB/s, done.
Resolving deltas: 100% (299/299), done.
Checking out files: 100% (186/186), done.
--------------




==============================================================================================================


With the fix in nfs-ganesha-2.7.3-9.el7rhgs.x86_64 build git clone was completed successfully-

v 4.0
-------------
[mani@f09-h03-000-1029u ganesha]$ git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
remote: Total 744 (delta 299), reused 669 (delta 224), pack-reused 0
Receiving objects: 100% (744/744), 5.00 MiB | 8.50 MiB/s, done.
Resolving deltas: 100% (299/299), done.
----------------

v 4.1
---------------
bash-4.2$ git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
remote: Total 744 (delta 299), reused 669 (delta 224), pack-reused 0
Receiving objects: 100% (744/744), 5.00 MiB | 0 bytes/s, done.
Resolving deltas: 100% (299/299), done.
----------------

v 3.0
---------------
bash-4.2$  git clone https://github.com/kasemir/pvws.git
Cloning into 'pvws'...
remote: Enumerating objects: 744, done.
remote: Counting objects: 100% (744/744), done.
remote: Compressing objects: 100% (482/482), done.
remote: Total 744 (delta 299), reused 669 (delta 224), pack-reused 0
Receiving objects: 100% (744/744), 5.00 MiB | 7.68 MiB/s, done.
Resolving deltas: 100% (299/299), done.
-----------------

Rest of the functionality cases will be run as part of regression cycle. Moving this BZ to verified state
Comment 24 Jiffin 2019-10-14 18:08:45 UTC 
LGTM
Comment 26 errata-xmlrpc 2019-10-30 12:15:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3252