Bug 1845166 - [Tracker for bug 1865762][RHEL][IBM] CephFS Volume has Permission Denied error while mounting
Summary: [Tracker for bug 1865762][RHEL][IBM] CephFS Volume has Permission Denied erro...
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: ocs-operator
Version: 4.3
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Jose A. Rivera
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On: 1865762
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-08 14:58 UTC by akgunjal@in.ibm.com
Modified: 2022-02-22 15:47 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1865762 (view as bug list)
Environment:
Last Closed: 2021-01-29 06:59:35 UTC
Embargoed:


Attachments (Terms of Use)

Description akgunjal@in.ibm.com 2020-06-08 14:58:49 UTC
Description of problem (please be detailed as possible and provide log
snippests):

CephFS Volume throws Permission Denied error when accessing the volume from pod.

I created a CephFS based PVC on the OpenShift cluster OCP 4.3 version. When I create a pod and mount it in the pod I get Permission Denied error while trying to access the mount path. It works fine when I create a privileged container. What is the way to use the CephFS volume using non-privileged container.


Version of all relevant components (if applicable):
Openshift OCP 4.3
OCS 4.3


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes. We cannot use CephFS volumes.


Is there any workaround available to the best of your knowledge?
After discussing in Ceph community, I enabled the container_use_cephfs SELinux boolean and then it started working in openshift. However, I am checking if its fine to enable this config from security point of view for customers in the production. And if this is the correct approach, will this be enabled by default on RHEL nodes.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
NA

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy OCS 4.3 and create a cephFS based PVC
2.Deploy a pod using cephFS volume
3.Exec inside the pod and navigate to mount path. Run "ls" command and you get "Permission Denied" error.


Actual results:
Run "ls" command and you get "Permission Denied" error.

Expected results:
You should be able to access mount path and read/write data to volume.

Additional info:
We have a GHE open for this issue https://github.com/ceph/ceph-csi/issues/1097

Comment 2 akgunjal@in.ibm.com 2020-06-08 15:02:08 UTC
I got one old BZ issue https://bugzilla.redhat.com/show_bug.cgi?id=1694045 related to the container_use_cephfs SELinux boolean setting. Adding here as reference to evaluate if thats the right way from security point of view.

Comment 3 Elad 2020-06-08 15:08:40 UTC
Hi,

Have you followed [1] and persistently enabled the container use of the Ceph file system in SELinux?

i.e:
# setsebool -P container_use_cephfs on



[1] https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html/deploying_openshift_container_storage/deploying-openshift-container-storage#enabling-file-system-access-for-containers-on-red-hat-enterprise-linux-based-nodes_rhocs

Comment 4 Neha Berry 2020-06-08 18:04:45 UTC

https://bugzilla.redhat.com/show_bug.cgi?id=1694045
https://bugzilla.redhat.com/show_bug.cgi?id=1692369(In reply to akgunjal.com from comment #2)
> I got one old BZ issue https://bugzilla.redhat.com/show_bug.cgi?id=1694045
> related to the container_use_cephfs SELinux boolean setting. Adding here as
> reference to evaluate if thats the right way from security point of view.

This selinux setting for cephfs is not enabled by default in RHEL as of yet.

We have had OCS bugs for similar problems in the past - e.g. Bug 177738 & Bug 1842518 and have requested for enabling this property by default . But, due to some limitations, this has not yet made part of install workflow of OCS/RHEL - more discussions could be found in the above referenced BZs. 

@humble, one more instance of problem due to "container_use_cephfs --> off" in RHEL , could you please re-confirm with RHEL/OCP/OCS stakeholders, whether we could somehow enable it in the install workflow itself ? Thanks again.


For RHCOS, following bugs were fixed to enable container_use_cephfs --> on . Do we have any such tracker bZ for RHEL/OCP ?

Comment 5 Humble Chirammal 2020-06-09 08:49:01 UTC
(In reply to Neha Berry from comment #4)
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1694045
> https://bugzilla.redhat.com/show_bug.cgi?id=1692369(In reply to
> akgunjal.com from comment #2)
> > I got one old BZ issue https://bugzilla.redhat.com/show_bug.cgi?id=1694045
> > related to the container_use_cephfs SELinux boolean setting. Adding here as
> > reference to evaluate if thats the right way from security point of view.
> 
> This selinux setting for cephfs is not enabled by default in RHEL as of yet.
> 
> We have had OCS bugs for similar problems in the past - e.g. Bug 177738 &
> Bug 1842518 and have requested for enabling this property by default . But,
> due to some limitations, this has not yet made part of install workflow of
> OCS/RHEL - more discussions could be found in the above referenced BZs. 

Jose/Umanga, Can we do anything here to make sure we enable this flag while installing OCS.
If yes, we have to prioritize this considering we are seeing more issues just because its not enabled in RHEL systems.

Comment 7 Jose A. Rivera 2020-07-01 14:00:21 UTC
Umanga is right, there's nothing that we can do here. We can't (and should not) alter host-level configuration.

Comment 8 Jose A. Rivera 2020-07-01 14:01:07 UTC
Keeping htis open to track any potential progres, but moving it out to OCS 4.6

Comment 9 Sahina Bose 2020-07-13 11:20:04 UTC
(In reply to Jose A. Rivera from comment #8)
> Keeping htis open to track any potential progres, but moving it out to OCS
> 4.6

Do we have any bug in OCP or RHEL to set the default for container_use_cephfs to on?

Comment 10 Sahina Bose 2020-07-13 11:43:59 UTC
Follow up on https://bugzilla.redhat.com/show_bug.cgi?id=1842518#c16, did we log a request to set defaults on RHEL8?

Comment 11 Humble Chirammal 2020-07-13 12:08:15 UTC
(In reply to Sahina Bose from comment #10)
> Follow up on https://bugzilla.redhat.com/show_bug.cgi?id=1842518#c16, did we
> log a request to set defaults on RHEL8?

bz#1842518 was closed and the attempt was figuring out a solution with OCS operator. Looks like thats not going to happen either.
The only option left here is giving a try or  request for RHEL enablement. 
Sahina, is it something we could do from OCS or RHCS team has to push the request?

Comment 12 Sahina Bose 2020-07-16 07:37:32 UTC
(In reply to Humble Chirammal from comment #11)
> (In reply to Sahina Bose from comment #10)
> > Follow up on https://bugzilla.redhat.com/show_bug.cgi?id=1842518#c16, did we
> > log a request to set defaults on RHEL8?
> 
> bz#1842518 was closed and the attempt was figuring out a solution with OCS
> operator. Looks like thats not going to happen either.
> The only option left here is giving a try or  request for RHEL enablement. 
> Sahina, is it something we could do from OCS or RHCS team has to push the
> request?

Yes - please open a bug in RHEL to add default

Comment 13 Humble Chirammal 2020-10-05 10:46:57 UTC
(In reply to Sahina Bose from comment #12)
> (In reply to Humble Chirammal from comment #11)
> > (In reply to Sahina Bose from comment #10)
> > > Follow up on https://bugzilla.redhat.com/show_bug.cgi?id=1842518#c16, did we
> > > log a request to set defaults on RHEL8?
> > 
> > bz#1842518 was closed and the attempt was figuring out a solution with OCS
> > operator. Looks like thats not going to happen either.
> > The only option left here is giving a try or  request for RHEL enablement. 
> > Sahina, is it something we could do from OCS or RHCS team has to push the
> > request?
> 
> Yes - please open a bug in RHEL to add default

Clearing needinfo as this request is tracked under https://bugzilla.redhat.com/show_bug.cgi?id=1865762

Comment 14 Jose A. Rivera 2020-10-06 14:44:19 UTC
This was erroneously moved to ASSIGNED, moving back to NEW.

Since we are dependent on RHEL for a fix, we won't be able to make it for this release. Moving to OCS 4.7.

Comment 15 Sahina Bose 2021-01-29 06:59:35 UTC
IBM ROKS deployer enables the sebool for "container_use_cephfs" on the worker nodes. Closing this as the RHEL bug was closed, and there's not much that we can do in OCS.


Note You need to log in before you can comment on or make changes to this bug.