Bug 1904915

Summary:	Creating CI/CD solution using OpenShift pipeline operator fails with NFS PV.
Product:	Red Hat OpenShift Pipelines	Reporter:	aghadge
Component:	pipelines	Assignee:	Vincent Demeester <vdemeest>
Status:	NEW ---	QA Contact:	Ruchir Garg <rgarg>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	aos-bugs, jsafrane
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description aghadge 2020-12-07 05:07:49 UTC

Description of problem:
-----------------------
Creating CI/CD solution using OpenShift pipeline operator fails with NFS PV.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
OCP v4.5

How reproducible:
----------------
Always


Steps to Reproduce:
-------------------
- Install the pipeline operator using steps mentioned below. 
 
   https://docs.openshift.com/container-platform/4.5/pipelines/installing-pipelines.html 

- Then for testing purpose tried to create a CI/CD solution using the steps mentioned in below documentation link.

  https://docs.openshift.com/container-platform/4.5/pipelines/creating-applications-with-cicd-pipelines.html

- When NFS PV is used for CI/CD solution then pipeline start fails with the error mentioned below.

STEP-CLONE

+ CHECKOUT_DIR=/workspace/output/
+ '[[' true '==' true ]]
+ cleandir
+ '[[' -d /workspace/output/ ]]
+ rm -rf /workspace/output//Dockerfile /workspace/output//README.md /workspace/output//go.mod /workspace/output//go.sum /workspace/output//image-digest /workspace/output//k8s /workspace/output//main.go /workspace/output//vendor
rm: can't remove '/workspace/output//image-digest': Operation not permitted

- Looks like NFS volume security is denying write access over the NFS volume and we might need to make use of supplemental groups or RunAsUser under SecurityContext to grant write access over the NFS PV.

- The particular PV is required to be mounted by "build-and-deploy" pods and all these pods are for running a particular job and which I guess is controlled by operator itself.

- I was not able to find any way to make this configuration changes for NFS PV to make it usable with pipeline.

- Need to know if there is any way we can incorporate NFS PV security related changes in pipeline pod to overcome this issue? If not then is it correct to consider that NFS PVs can not be used with CI/CD solutions at least for testing purpose.

Actual results:
---------------
- Openshift Pipeline start fails due to access issues with NFS PV

Expected results:
-----------------
- Openshift Pipeline should start without any issues.

Additional info:
----------------
- I also tested this with dynamic provisioning where volume was provisioned from "gce-pd" provisioner over GCP and i did not face any issue there.

Comment 1 Jan Safranek 2020-12-07 13:36:33 UTC

Shared storage like NFS is tricky, OCP cannot use Pod's `fsGroup` to make sure a pod can access the volume. You can either use supplemental groups or user IDs, both described at https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/persistent-storage-nfs.html#nfs-volume-security_persistent-storage-nfs. I'm assigning this to Tekton team if they can suggest how to pass supplemental group to Pods that use PVs or if they're willing to take a RFE or declare NFS and other shared storage unsupported.