Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1494369

Summary: nfs share gets wrong permissions if subdir feature is used.
Product: OpenShift Container Platform Reporter: Alexander Koksharov <akokshar>
Component: StorageAssignee: Tomas Smetana <tsmetana>
Status: CLOSED ERRATA QA Contact: Wenqi He <wehe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: akokshar, aos-bugs, aos-storage-staff, bchilds, eparis, hchen, nnosenzo
Target Milestone: ---Keywords: Reopened
Target Release: 3.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: In case NFS and subdir option are used docker creates new directory with incorrect access rights. Consequence: Pod cannot write into the created directory. Fix: OpenShift creates the new directory with correct permissions. Result: Pod can write into the created directory.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-18 07:00:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Koksharov 2017-09-22 06:52:36 UTC
Description of problem:

- nfs is being used as persistent storage. share configured with 'rw,root_squash' options
- "subdir" option is set in dc for a mount
- openshift creates subdir path and do successful mount to a pod
- directory created by openshift gets permissions 755 and ownership nfsnobody:nfsnobody.
- pod can not write to the share. it has nfsnobody as a supplementary group but writes are not allowed for a group members.
- there are file ACL configured on shared directory to inherit permissions but these still get changed by client's umask, so effective permissions are still 755

During the tests it was discovered that latest 7.4 rhel, together with nfsmount with nfs vers=4.2 do not respect client's umask.
However, nfs4.1 and 4.0 are enforcing umask breaking openshift application.

Nfs 4.2 seems to be not yet supported.

Do you see any side effects of changing openshift node plroceses umask to 0002?
Is safe to update systemd unit and set ExecStart=umask=0002;/usr/bin/openshift .....

Version-Release number of selected component (if applicable):
v3.4.1.44

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 hchen 2017-09-22 12:51:30 UTC
We should not support NFS subdir. NFS volume doesn't support ownership management and thus should not create subdir or chown.

Comment 2 Alexander Koksharov 2017-10-02 08:20:31 UTC
Hello,

The problem is not on NFS side. The problem is not an ownership of a subdirectories.

The problem is that umask configured on a node is impacting application. This, in my opinion, should not be the case. There should be the possibility to define custom umask for a subdir. So, that administrator can control what permission bits will be set for a new directory created.
I did not test, but I suspect there is same behaviour/issue with all kinds of PVs that are baked by 'filesystem' (not 'block') type of storage.

Comment 3 Alexander Koksharov 2017-10-02 08:25:53 UTC
Please also advise whether changing atomic-openshift-node.service systemd unit wont case other problems. As a workaround for a customer I consider change of unit as this: 
...
ExecStart=umask=0002;/usr/bin/openshift ...
...

But, I need your expert advise here. Will such a change cause any other issues?

Comment 4 Bradley Childs 2017-10-05 22:21:26 UTC
This sounds like a umask/node configuration issue and not related to openshift storage.

Personally I dont know if modifying atomic-openshift-node.service as mentioned would have an impact...  The only way would be to try and see what happens.  You may ask this question to the aos-devel mailing list where it should get a better response.


Closing as i dont see any Openshift bug/issue.

Comment 5 Alexander Koksharov 2017-11-02 08:18:19 UTC
Hello Bradley,

I am reopening this bz since I cant get answer from aos-devel/openshift-sme mailing lists.

This issue us a bug since there is not mentions that subdir feature does not work for NFS.

Please provide me with a workaround/fix so I can advise Customer on how to move forward.

Comment 6 Alexander Koksharov 2017-11-02 15:11:54 UTC
To add some more input to support my statement that this is a bug:
Web-Console does present an option to set subpath (dc config -> add storage).
So, enduser does not care what baking storage is. Option is presented and it is expected to be working. 

we did capture network trace and clearly seen that umask 022 is requested. Which is incorrect, since pod later can not write to the created folder. The fact that pod have correct supplementary group does not matter - it still have no write permission.

Comment 7 Bradley Childs 2017-11-02 18:46:32 UTC
Alex - Can you provide a dump of the deployment config being used

Comment 9 Alexander Koksharov 2017-11-06 08:57:25 UTC
Hello Pavel,

Thank you for your verifications. 
Do you have any estimation about when this fix can be available to the customer?

Is there a possibility to share updated binaries so customer can test fix by them selves?

--
Alexander

Comment 20 Nicolas Nosenzo 2017-12-12 16:28:58 UTC
@Pavel,
Has PR [0] been included on any release yet ?

[0] https://github.com/openshift/ose/pull/919

Comment 25 Wenqi He 2018-01-25 03:51:07 UTC
Tested on below version:
openshift v3.4.1.44.49
kubernetes v1.4.0+776c994

The mentioned fix in bug #1445523 has been fixed in 3.4

# oc exec -it subpath sh
/ $ ls -ld /mnt/direct/
drwxrwsrwx    2 root     users            6 Jan 25 03:47 /mnt/direct/
/ $ ls -ld /mnt/subpath/
drwxrwsrwx    2 root     users            6 Jan 25 03:47 /mnt/subpath/
/ $ touch /mnt/subpath/testfile
/ $ exit

Comment 29 errata-xmlrpc 2018-04-18 07:00:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1134