Description of problem:
Cu is using EBS as backend storage for pods, pods which are creating with persistent storage, the volume gets mounted inside the pod but the data is not writable.
Cu sees permission denied error on the pods logs and the pods move to crashloopbackoff.


Version-Release number of selected component (if applicable):
3.7

How reproducible:
Unconfirmed

Steps to Reproduce:
Cu deployed 'jenkin-persistent' with pvc in dynamic provisioned storageclass for test, fails. Does not works with hawkular as well, and any other apps.

Actual results:
=> sourcing 10-check-env-vars.sh ...
=> sourcing 20-setup-wiredtiger-cache.sh ...
=> [Tue Mar 20 18:29:44] wiredTiger cacheSizeGB set to 1
=> sourcing 30-set-config-file.sh ...
=> sourcing 35-setup-default-datadir.sh ...
ERROR: Couldn't write into /var/lib/mongodb/data
CAUSE: current user doesn't have permissions for writing to /var/lib/mongodb/data directory
DETAILS: current user id = 184, user groups: 997 0
DETAILS: directory permissions: drwxr-xr-x owned by 0:0, SELinux: system_u:object_r:svirt_sandbox_file_t:s0:c5,c11

Expected results:
Write to work as before


PV Dump:
# oc get pv pvc-217cf5bf-2c7a-11e8-a658-0aab460a5b68 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: aws-ebs-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
  creationTimestamp: 2018-03-20T20:06:10Z
  labels:
    failure-domain.beta.kubernetes.io/region: us-west-2
    failure-domain.beta.kubernetes.io/zone: us-west-2a
  name: pvc-217cf5bf-2c7a-11e8-a658-0aab460a5b68
  resourceVersion: "7425311"
  selfLink: /api/v1/persistentvolumes/pvc-217cf5bf-2c7a-11e8-a658-0aab460a5b68
  uid: 2213e1a2-2c7a-11e8-a658-0aab460a5b68
spec:
  accessModes:
  - ReadWriteOnce
  awsElasticBlockStore:
    fsType: ext4
    volumeID: aws://us-west-2a/vol-xxxxxxxxxxxxx
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: mongodb
    namespace: se-playground
    resourceVersion: "7425288"
    uid: 217cf5bf-2c7a-11e8-a658-0aab460a5b68
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gp2
status:
  phase: Bound


# oc get pvc mongodb -o yaml -n se-playground
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
  creationTimestamp: 2018-03-20T20:06:09Z
  labels:
    template: mongodb-persistent-template
  name: mongodb
  namespace: se-playground
  ownerReferences:
  - apiVersion: template.openshift.io/v1
    blockOwnerDeletion: true
    kind: TemplateInstance
    name: ddbbeaab-bec4-41f0-aef5-67bf16a47522
    uid: 21741114-2c7a-11e8-a658-0aab460a5b68
  resourceVersion: "7425313"
  selfLink: /api/v1/namespaces/se-playground/persistentvolumeclaims/mongodb
  uid: 217cf5bf-2c7a-11e8-a658-0aab460a5b68
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp2
  volumeName: pvc-217cf5bf-2c7a-11e8-a658-0aab460a5b68
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound


StorageClass Dump (if StorageClass used by PV/PVC):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-02-23T17:44:45Z
  name: gp2
  resourceVersion: "7814"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: 3c9babb6-18c1-11e8-a98c-xxxxxxxxxxx
parameters:
  encrypted: "false"
  kmsKeyId: ""
  type: gp2
provisioner: kubernetes.io/aws-ebs


Additional info:
-The persistent volume and persistentVolumeClaims are able to bound each other, 
-Deployed a ephemeral jenkin for test, the pods came up but later attached persistent storage, but still did not work and we saw permission denied errors on it.
-This was previously working as of last week. Customer updated packages and it seems that shortly after it failed. We ran through the proper, full upgrade process and issue continued.