Description of problem:
Unable to write to the container mount path of cephfs volume, got "Input/output error"


Version-Release number of selected component (if applicable):
openshift v3.10.0-0.16.0
kubernetes v1.9.1+a0ce1bc657

How reproducible:
Always

Steps to Reproduce:
1. Create CephFS storageclass, PVC. PV is provisioned
2. Create Pod
3. oc rsh pod, try creating a file in the mount path

Actual results:
/mnt/cephfs # touch ls
touch: ls: Input/output error

/mnt/cephfs # dmesg |tail
[1056542.414236] libceph: mon0 xxx:6789 session established
[1056542.415566] libceph: client14176 fsid 310b6b01-99fe-43fb-88f5-efaa9317515e
[1056542.599772] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)

Expected results:
Do not see "Input/output error"

Master Log:

Node Log (of failed PODs):

PV Dump:
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    cephFSProvisionerIdentity: ceph.com/cephfs
    cephShare: kubernetes-dynamic-pvc-bc6d0ed5-3c94-11e8-a12a-0a580a800010
    pv.kubernetes.io/provisioned-by: ceph.com/cephfs
  creationTimestamp: null
  name: pvc-bc5a83e5-3c94-11e8-a6af-0050569f41c6
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  cephfs:
    monitors:
    - xxx:6789
    path: /volumes/kubernetes/kubernetes-dynamic-pvc-bc6d0ed5-3c94-11e8-a12a-0a580a800010
    secretRef:
      name: ceph-kubernetes-dynamic-user-bc6d0f82-3c94-11e8-a12a-0a580a800010-secret
      namespace: jhou
    user: kubernetes-dynamic-user-bc6d0f82-3c94-11e8-a12a-0a580a800010
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: cephfsc
    namespace: jhou
    resourceVersion: "1474926"
    uid: bc5a83e5-3c94-11e8-a6af-0050569f41c6
  persistentVolumeReclaimPolicy: Delete
  storageClassName: cephfs
status: {}


PVC Dump:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"18d7c220-3c8d-11e8-a12a-0a580a800010","leaseDurationSeconds":15,"acquireTime":"2018-04-10T07:56:54Z","renewTime":"2018-04-10T07:56:56Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-class: cephfs
    volume.beta.kubernetes.io/storage-provisioner: ceph.com/cephfs
  creationTimestamp: null
  finalizers:
  - kubernetes.io/pvc-protection
  name: cephfsc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-bc5a83e5-3c94-11e8-a6af-0050569f41c6
status: {}


StorageClass Dump (if StorageClass used by PV/PVC):
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: null
  name: cephfs
parameters:
  adminId: admin
  adminSecretName: cephrbd-secret
  adminSecretNamespace: default
  monitors: xxx:6789
provisioner: ceph.com/cephfs
reclaimPolicy: Delete
volumeBindingMode: Immediate


Additional info:
# ceph auth ls
...
client.kubernetes-dynamic-user-bc6d0f82-3c94-11e8-a12a-0a580a800010
        key: AQBHbsxaitnwBBAA/ST6a9SU2bm8MTr9HNvTuw==
        caps: [mds] allow r,allow rw path=/volumes/kubernetes/kubernetes-dynamic-pvc-bc6d0ed5-3c94-11e8-a12a-0a580a800010
        caps: [mon] allow r
        caps: [osd] allow rw pool=cephfs_data namespace=fsvolumens_kubernetes-dynamic-pvc-bc6d0ed5-3c94-11e8-a12a-0a580a800010