Bug 1293805

Summary: Default SCC forbid recycler pod to be create, cause Persistent Volume failed to recycle
Product: OpenShift Container Platform Reporter: Liang Xia <lxia>
Component: StorageAssignee: Mark Turansky <mturansk>
Status: CLOSED ERRATA QA Contact: Liang Xia <lxia>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.0CC: aos-bugs, bchilds, bkozdemb, clasohm, deads, ederevea, erich, haowang, jkrieger, jswensso, knakayam, mwysocki, nicholas_schuetz, nmunnell, pep, rhowe, tdawson
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-12 16:26:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1267746    

Description Liang Xia 2015-12-23 07:03:56 UTC
Description of problem:
Default SCC forbidden recycler pod to be create,
thus Persistent Volume can not be recycle.

Version-Release number of selected component (if applicable):
openshift v3.1.1.0
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

How reproducible:
Always

Steps to Reproduce:
1.Create PV (persistent volume) with reclaim policy "Recycle"
2.Create PVC (persistent volume claim)
3.Create pod use above pvc.
4.Delete pod and pvc.
5.Check PV status.

Actual results:
# oc describe pv nfs
Name:        nfs
Labels:        <none>
Status:        Failed
Claim:        lxiap/nfsc
Reclaim Policy:    Recycle
Access Modes:    RWO
Capacity:    5Gi
Message:    Recycling error: Unexpected error creating recycler pod:  Pod "pv-recycler-nfs-" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.containers[0].securityContext.securityContext.runAsUser: invalid value '0', Details: UID on container pv-recycler does not match required range.  Found 0, required min: 1000000000 max: 1000009999]
Source:
    Type:    NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    10.240.0.5
    Path:    /data
    ReadOnly:    false


Expected results:
Default SCC should allow recycler pod creating/running.
PV should be Available after recycler pod finished its job.

Additional info:
# oc get scc
NAME               PRIV      CAPS      HOSTDIR   SELINUX     RUNASUSER          FSGROUP    SUPGROUP   PRIORITY
anyuid             false     []        false     MustRunAs   RunAsAny           RunAsAny   RunAsAny   10
hostaccess         false     []        true      MustRunAs   MustRunAsRange     RunAsAny   RunAsAny   <none>
hostmount-anyuid   false     []        true      MustRunAs   RunAsAny           RunAsAny   RunAsAny   <none>
nonroot            false     []        false     MustRunAs   MustRunAsNonRoot   RunAsAny   RunAsAny   <none>
privileged         true      []        true      RunAsAny    RunAsAny           RunAsAny   RunAsAny   <none>
restricted         false     []        false     MustRunAs   MustRunAsRange     RunAsAny   RunAsAny   <none>

Comment 1 David Eads 2016-01-27 14:47:06 UTC
@markturansky Looks like you forgot to add some SCC powers for the recycler SA.  You'll want to document what you think you need when you add it here: https://github.com/openshift/origin/blob/master/pkg/cmd/server/bootstrappolicy/securitycontextconstraints.go#L236-L250

Comment 2 Mark Turansky 2016-01-28 17:11:53 UTC
Fixed in https://github.com/openshift/origin/pull/6884

Comment 3 Kenjiro Nakayama 2016-02-09 01:03:42 UTC
How is the progress of this ticket? 

  https://github.com/openshift/origin/pull/6884 is still not merged.

One of the enterprise customers hit this issue, and we need the fix as soon as possible.

Comment 6 Ryan Howe 2016-02-09 20:01:29 UTC
Current workaround for v3.1.1.6 


1. SA "pv-recycler-controller" should already be created and located in openshift-infra namespace [1]: 

  $ oc get sa -n openshift-infra 

2. If this service account is not added we will need to add it manually [2]. 

3. Add the the service account to the SCC hostmount-anyuid

  $ oadm policy add-scc-to-user hostmount-anyuid system:serviceaccount:openshift-infra:pv-recycler-controller 


*NOTE: Persistent Volumes in already in a failed state will not recover and will need to be deleted and added back to the environment. The contents will also need to be manually scrubbed. 

[1]
~~~
┌─[root@master1]─[~]
└──> oc get sa -n openshift-infra 
NAME                        SECRETS   AGE
build-controller            2         24d
builder                     3         24d
default                     4         24d
deployer                    2         24d
deployment-controller       2         24d
hpa-controller              2         24d
job-controller              3         24d
pv-binder-controller        3         13d
pv-controller               2         24d
pv-provisioner-controller   2         13d
pv-recycler-controller      2         13d
replication-controller      2         24d
~~~

[2]
~~~
┌─[root@master1]─[~]
└──> oc create -n openshift-infra -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pv-recycler-controller
API
~~~

Comment 8 Mark Turansky 2016-02-17 14:16:04 UTC
The fix has merged: https://github.com/openshift/origin/pull/6884

Comment 9 Liang Xia 2016-02-22 05:46:31 UTC
Verified on version,
$ openshift version
openshift v3.1.1.904
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5

Persistent Volumes can be recycled now.

Comment 14 errata-xmlrpc 2016-05-12 16:26:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064