Bug 1324418

Could you supply the output of `oc get pod <name of recycler pod> -o yaml`?  It looks like the recycler pod was matched against an SCC that didn't allow use of uid 0 or the nfs volume plugin.

This error is from pv after the pod and pvc is deleted, and should wait 1 hour after re-create the pvc.
The pod I created is like below:

`oc get pod nfs -o yaml`

apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/scc: anyuid
  creationTimestamp: 2016-04-07T01:58:09Z
  labels:
    name: frontendhttp
  name: nfs
  namespace: default
  resourceVersion: "8500"
  selfLink: /api/v1/namespaces/default/pods/nfs
  uid: 2d42ac10-fc64-11e5-a73f-6c0b849ad3e9
spec:
  containers:
  - image: aosqe/hello-openshift
    imagePullPolicy: IfNotPresent
    name: myfrontend
    ports:
    - containerPort: 80
      name: http-server
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        drop:
        - KILL
        - MKNOD
        - SYS_CHROOT
      privileged: false
      seLinuxOptions:
        level: s0:c5,c0
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /mnt/nfs
      name: pvol
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-esl5w
      readOnly: true
  dnsPolicy: ClusterFirst
  host: dhcp-137-221.nay.redhat.com
  imagePullSecrets:
  - name: default-dockercfg-1vrdk
  nodeName: dhcp-137-221.nay.redhat.com
  restartPolicy: Always
  securityContext:
    seLinuxOptions:
      level: s0:c5,c0
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - name: pvol
    persistentVolumeClaim:
      claimName: nfsc
  - name: default-token-esl5w
    secret:
      secretName: default-token-esl5w
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2016-04-07T01:58:13Z
    status: "True"
    type: Ready
  containerStatuses:
  - containerID: docker://802783f457dd085d434472281b525436efa180932d1acb3d01aa5eddc3fcb56f
    image: aosqe/hello-openshift
    imageID: docker://cddcd4ab363acd31256ed7880d4b669fa45227e49eec41429f80a4f252dfb0da
    lastState: {}
    name: myfrontend
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2016-04-07T01:58:13Z
  hostIP: 10.66.136.120
  phase: Running
  podIP: 172.17.0.2
  startTime: 2016-04-07T01:58:09Z

`oc get scc`

anyuid             false     []        MustRunAs   RunAsAny           RunAsAny    RunAsAny    10         false            [configMap downwardAPI emptyDir persistentVolumeClaim secret]
hostaccess         false     []        MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    <none>     false            [configMap downwardAPI emptyDir hostPath persistentVolumeClaim secret]
hostmount-anyuid   false     []        MustRunAs   RunAsAny           RunAsAny    RunAsAny    <none>     false            [configMap downwardAPI emptyDir hostPath persistentVolumeClaim secret]
hostnetwork        false     []        MustRunAs   MustRunAsRange     MustRunAs   MustRunAs   <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim secret]
nonroot            false     []        MustRunAs   MustRunAsNonRoot   RunAsAny    RunAsAny    <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim secret]
privileged         true      []        RunAsAny    RunAsAny           RunAsAny    RunAsAny    <none>     false            [*]
restricted         false     []        MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim secret]

`oc get pv nfs -o yaml`

apiVersion: v1
kind: PersistentVolume
metadata:
  creationTimestamp: 2016-04-07T01:57:53Z
  name: nfs
  resourceVersion: "8490"
  selfLink: /api/v1/persistentvolumes/nfs
  uid: 24100fc0-fc64-11e5-a73f-6c0b849ad3e9
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: nfsc
    namespace: default
    resourceVersion: "8486"
    uid: 280074db-fc64-11e5-a73f-6c0b849ad3e9
  nfs:
    path: /mnt/nfs
    server: <ip addr of nfs server>
  persistentVolumeReclaimPolicy: Recycle
status:
  phase: Bound

(In reply to Paul Morie from comment #1)
> Could you supply the output of `oc get pod <name of recycler pod> -o yaml`? 
> It looks like the recycler pod was matched against an SCC that didn't allow
> use of uid 0 or the nfs volume plugin.

Yes, I think so, the pv recycler pod wasn't created because SCC constrained it. I tried to work around this issue by updating restricted SCC, then the issue disappeared. My updated SCC was:

```
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- hostPath
- nfs
- persistentVolumeClaim
- secret
```

> Yes, I think so, the pv recycler pod wasn't created because SCC constrained
> it. I tried to work around this issue by updating restricted SCC, then the
> issue disappeared. 
 
Can this bug be closed then or is there still an outstanding issue ?

I'm wondering which serviceaccount the recycler pod was being run under.

Can you repeat this test with --loglevel=4 and post the API server log?

We are looking for a message like 'getting security context constraints for pod'.

Here is the information I search from docker logs:
I0408 02:31:12.495997    2597 admission.go:109] getting security context constraints for pod  (generate: pv-recycler-nfs-) in namespace openshift-infra with user info &{system:serviceaccount:openshift-infra:pv-recycler-controller 5e78dce3-fd2c-11e5-ae17-6c0b849ad3e9 [system:serviceaccounts system:serviceaccounts:openshift-infra system:authenticated]}
I0408 02:31:12.496020    2597 admission.go:118] getting security context constraints for pod  (generate: pv-recycler-nfs-) with service account info &{system:serviceaccount:openshift-infra:default  [system:serviceaccounts system:serviceaccounts:openshift-infra]}

Tried to re-pull the latest image and test, still repro

The problem is that the hostmount-anyuid SCC, which the PV recycler controller service account is supposed to use, doesn't have access to the nfs plugin:

From the controller manager log:

408 13:53:56.948588   31157 admission.go:109] getting security context constraints for pod  (generate: pv-recycler-nfs-) in namespace openshift-infra with user info &{system:serviceaccount:openshift-infra:pv-r
ecycler-controller d1efb715-fdb2-11e5-8f80-68f728db1985 [system:serviceaccounts system:serviceaccounts:openshift-infra system:authenticated]}
I0408 13:53:56.948642   31157 admission.go:118] getting security context constraints for pod  (generate: pv-recycler-nfs-) with service account info &{system:serviceaccount:openshift-infra:default  [system:servi
ceaccounts system:serviceaccounts:openshift-infra]}
I0408 13:53:56.949928   31157 admission.go:368] got preallocated values for min: 1000020000, max: 1000029999 for uid range in namespace openshift-infra
I0408 13:53:56.950211   31157 admission.go:381] got preallocated value for level: s0:c5,c0 for selinux options in namespace openshift-infra
I0408 13:53:56.950227   31157 admission.go:411] got preallocated value for groups: 1000020000/10000 in namespace openshift-infra
I0408 13:53:56.950268   31157 admission.go:381] got preallocated value for level: s0:c5,c0 for selinux options in namespace openshift-infra
I0408 13:53:56.950286   31157 admission.go:525] validating pod  (generate: pv-recycler-nfs-) against providers restricted,hostmount-anyuid
I0408 13:53:56.950355   31157 admission.go:154] unable to validate pod  (generate: pv-recycler-nfs-) against any security context constraint: [securityContext.runAsUser: Invalid value: 0: UID on container pv-rec
ycler does not match required range.  Found 0, required min: 1000020000 max: 1000029999 spec.containers[0].securityContext.volumes[0]: Invalid value: "nfs": nfs volumes are not allowed to be used spec.containers
[0].securityContext.volumes[0]: Invalid value: "nfs": nfs volumes are not allowed to be used]
E0408 13:53:56.979235   31157 persistentvolume_recycler_controller.go:245] PersistentVolume[test-pv] failed recycling: Unexpected error creating recycler pod:  pods "pv-recycler-nfs-" is forbidden: unable to val
idate against any security context constraint: [securityContext.runAsUser: Invalid value: 0: UID on container pv-recycler does not match required range.  Found 0, required min: 1000020000 max: 1000029999 spec.co
ntainers[0].securityContext.volumes[0]: Invalid value: "nfs": nfs volumes are not allowed to be used spec.containers[0].securityContext.volumes[0]: Invalid value: "nfs": nfs volumes are not allowed to be used]

The problem is that the `hostmount-anyuid` SCC that the PV recycler is supposed to use doesn't have permission to use the NFS plugin:

$ oc get scc hostmount-anyuid -o yaml                                                                                                                                                                              
allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: v1
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: 'hostmount-anyuid provides all the features of the
      restricted SCC but allows host mounts and any UID by a pod.  This is primarily
      used by the persistent volume recycler. WARNING: this SCC allows host file system
      access as any UID, including UID 0.  Grant with caution.'
  creationTimestamp: 2016-04-08T17:44:09Z
  name: hostmount-anyuid
  resourceVersion: "75"
  selfLink: /api/v1/securitycontextconstraints/hostmount-anyuid
  uid: 7f85172c-fdb1-11e5-8b50-68f728db1985
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities: null
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:openshift-infra:pv-recycler-controller
volumes:
- configMap
- downwardAPI
- emptyDir
- hostPath
- persistentVolumeClaim
- secret

PR for the fix: https://github.com/openshift/origin/pull/8432

I have verified this on OSE and it is fixed below version:
openshift v3.2.0.15
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

Already verified from last comment.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064