Bug 1569359 - No write permission in a directory mounted as a PVC with Azure Disk without fsGroup
Summary: No write permission in a directory mounted as a PVC with Azure Disk without f...
Keywords:
Status: CLOSED DUPLICATE of bug 1559100
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Hemant Kumar
QA Contact: Wenqi He
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-19 07:05 UTC by Takayoshi Tanaka
Modified: 2018-04-25 02:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-25 01:33:57 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Takayoshi Tanaka 2018-04-19 07:05:24 UTC
Description of problem:
When using PVC with Azure Disk, the pod has no write permission in the directory mounted the PVC.

Version-Release number of selected component (if applicable):
# oc version
oc v3.9.14
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ose3-single-vm.westus2.cloudapp.azure.com:8443
openshift v3.9.14
kubernetes v1.9.1+a0ce1bc657


How reproducible:
Always

Steps to Reproduce:
1. Create a default storage class with Azure Disk dynamic provisioning
2. Create a PVC
3. Deploy a pod with PVC

Instead of 2 and 3, we can also see the same issue when deploying a mongodb template in the service catalog.

Actual results:
A pod has no write permission in the mounted directory. In the mongodb case, the deployment was failed because a pod can't start correctly due to insufficient permission.

Expected results:
A pod has a write permission in the mounted directory.

in the debug pod if mongodb:
$ whoami
mongodb
$ groups
mongodb root
$ ls -al /var/lib/mongodb/data
drwxr-xr-x. 3 root root 4096 Apr 19 04:38 .


PVC Dump:
# oc get pvc mongodb -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
  creationTimestamp: 2018-04-19T04:35:27Z
  labels:
    template: mongodb-persistent-template
  name: mongodb
  namespace: development
  ownerReferences:
  - apiVersion: template.openshift.io/v1
    blockOwnerDeletion: true
    kind: TemplateInstance
    name: a0c596cb-be7a-429f-ba2c-918f0aab763a
    uid: 153c5cac-438b-11e8-990d-000d3af9651a
  resourceVersion: "8553957"
  selfLink: /api/v1/namespaces/development/persistentvolumeclaims/mongodb
  uid: 157698b8-438b-11e8-990d-000d3af9651a
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: slow
  volumeName: pvc-157698b8-438b-11e8-990d-000d3af9651a
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound


StorageClass Dump (if StorageClass used by PV/PVC):
# oc get sc slow -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-01-15T07:00:08Z
  name: slow
  resourceVersion: "7393311"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/slow
  uid: b90952cf-f9c1-11e7-8ec2-000d3af9651a
parameters:
  kind: managed
  storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete


Additional info:
If I set the securityContext in the pod, a pod has a write permission. 

~~~
    spec:
      containers:
        ...
      securityContext:
        fsGroup: 1000140000
        runAsUser: 1000140000
~~~

drwxrwxr-x. 3 root 1000140000

It could be a workaround. However, I think the fix is required for the following reason.

- All existing deployment configs should be fixed to add securityContext.
- If the volume is shared across pods, all pods should have the same secuerityContext.
- A lot of service catalogs such as mongodb are not available by the default template. The users have to add securityContext.


I haven't tested completely but this issue is happening in the latest minor version of OCP 3.6 & 3.7.
I reported a similar issue with "Azure File".
https://access.redhat.com/solutions/3403731

However, I think it's not related because the mounting mechanism in kubernetes is quite different between Azure Disk and Azure File.

Comment 1 Takayoshi Tanaka 2018-04-19 07:08:42 UTC
The link is not BZ but our KCS. Here is the correct BZ link.
https://bugzilla.redhat.com/show_bug.cgi?id=1564976

Comment 2 Wenqi He 2018-04-23 06:14:36 UTC
My pod without fsGroup could be running and end user has the writing access, not sure whether this is the situation you reported.

# oc version
oc v3.9.24
kubernetes v1.9.1+a0ce1bc657

# oc get sc -o yaml
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    creationTimestamp: 2018-04-23T03:08:42Z
    name: sc-36ryt
    namespace: ""
    resourceVersion: "9170"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/sc-36ryt
    uid: a0e832a7-46a3-11e8-aefb-000d3a11d3cd
  parameters:
    kind: Dedicated
    storageaccount: eastusimg
  provisioner: kubernetes.io/azure-disk
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
azpvc     Bound     pvc-a257939b-46a3-11e8-aefb-000d3a11d3cd   1Gi        RWO            sc-36ryt       25m

# oc get pv pvc-a257939b-46a3-11e8-aefb-000d3a11d3cd -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
    volumehelper.VolumeDynamicallyCreatedByKey: azure-disk-dynamic-provisioner
  creationTimestamp: 2018-04-23T03:08:45Z
  name: pvc-a257939b-46a3-11e8-aefb-000d3a11d3cd
  resourceVersion: "9180"
  selfLink: /api/v1/persistentvolumes/pvc-a257939b-46a3-11e8-aefb-000d3a11d3cd
  uid: a26cb1fe-46a3-11e8-aefb-000d3a11d3cd
spec:
  accessModes:
  - ReadWriteOnce
  azureDisk:
    cachingMode: ReadWrite
    diskName: kubernetes-dynamic-pvc-a257939b-46a3-11e8-aefb-000d3a11d3cd
    diskURI: https://eastusimg.vhd
    fsType: ext4
    kind: Dedicated
    readOnly: false
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: azpvc
    namespace: 36ryt
    resourceVersion: "9175"
    uid: a257939b-46a3-11e8-aefb-000d3a11d3cd
  persistentVolumeReclaimPolicy: Delete
  storageClassName: sc-36ryt
  volumeMode: Filesystem
status:
  phase: Bound

$ oc get pods
NAME      READY     STATUS    RESTARTS   AGE
azpvcpo   1/1       Running   0          27m
$ oc exec -it azpvcpo sh
/ $ ls /mnt/azure/
ad-36ryt    lost+found
/ $ touch /mnt/azure/wehe
/ $ ls /mnt/azure/
ad-36ryt    lost+found  wehe
/ $ id
uid=1000150000 gid=0(root) groups=1000150000

Pod: https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/misc/pod.yaml

Comment 3 Wenqi He 2018-04-23 06:32:59 UTC
And I tried with mongodb-persistent
$ oc new-app mongodb-persistent
$ oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb   Bound     pvc-6c97bf73-46be-11e8-aefb-000d3a11d3cd   1Gi        RWO            azddef         5m
$ oc get pods
NAME              READY     STATUS    RESTARTS   AGE
mongodb-1-59cfn   1/1       Running   0          6m
$ oc get dc -o yaml
...
          securityContext:
            capabilities: {}
            privileged: false
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /var/lib/mongodb/data
            name: mongodb-data
...
$ oc exec -it mongodb-1-59cfn bash
bash-4.2$ id
uid=1000160000 gid=0(root) groups=0(root),1000160000
bash-4.2$ touch /var/lib/mongodb/data/wehe
bash-4.2$ exit

Comment 4 Takayoshi Tanaka 2018-04-23 07:34:43 UTC
Could you execute the following commands?

- in the pod
# ls -al /mnt/azure/

- in the node
# mount | grep pvc-a257939b-46a3-11e8-aefb-000d3a11d3cd
$ ls -al <mount_point_in_the_node>

Comment 5 Wenqi He 2018-04-23 09:13:50 UTC
$ ls -al /var/lib/mongodb/data/
total 284
drwxrwsr-x. 5 root       1000190000  4096 Apr 23 08:54 .
drwxrwxr-x. 3 mongodb    root          18 Apr  5 09:53 ..
-rw-r--r--. 1 1000190000 1000190000     0 Apr 23 08:54 wehe


# ls -al /var/lib/origin/openshift.local.volumes/pods/30f5b4b4-46d2-11e8-aefb-000d3a11d3cd/volumes/kubernetes.io~azure-disk/pvc-2eac81ba-46d2-11e8-aefb-000d3a11d3cd
total 284
drwxrwsr-x. 5 root       1000190000  4096 Apr 23 08:57 .
drwxr-x---. 3 root       root          54 Apr 23 08:44 ..
-rw-r--r--. 1 1000190000 1000190000     0 Apr 23 08:54 wehe

Seems fsGroup is automatically added by scc based on the project range
$ oc get pods mongodb-1-vxr8l -o yaml
...
    securityContext:
      privileged: false
      runAsUser: 1000190000
...
  securityContext:
    fsGroup: 1000190000
    seLinuxOptions:
      level: s0:c14,c4
...

$ oc get project -o yaml
apiVersion: v1
items:
- apiVersion: project.openshift.io/v1
  kind: Project
  metadata:
    annotations:
      openshift.io/description: ""
      openshift.io/display-name: ""
      openshift.io/requester: wehe
      openshift.io/sa.scc.mcs: s0:c14,c4
      openshift.io/sa.scc.supplemental-groups: 1000190000/10000
      openshift.io/sa.scc.uid-range: 1000190000/10000
    creationTimestamp: 2018-04-23T08:39:49Z
    name: wehe
    namespace: ""

Comment 6 Takayoshi Tanaka 2018-04-25 00:51:32 UTC
If the fsGroup is not added explicity or via project range, the mounted directiry is 0755 and pod has no write permission. Is it correct?

Comment 7 Hemant Kumar 2018-04-25 00:57:30 UTC
It could be because of SCC with which pod is running. Can you post full pod yaml in the output? The SCC will be an annotation. Once you know the SCC can you also post SCC's yaml via "oc get scc <name> -o json" ?

Comment 8 Takayoshi Tanaka 2018-04-25 01:24:17 UTC
I haven't used SCC. Here is a pod definition.

~~~
apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/deployment-config.latest-version: "1"
    openshift.io/deployment-config.name: httpd-test3
    openshift.io/deployment.name: httpd-test3-1
    openshift.io/scc: anyuid
  creationTimestamp: 2018-04-25T01:19:30Z
  generateName: httpd-test3-1-
  labels:
    deployment: httpd-test3-1
    deploymentconfig: httpd-test3
    name: httpd-test3
  name: httpd-test3-1-4zb4s
  namespace: development
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicationController
    name: httpd-test3-1
    uid: af85def3-4826-11e8-990d-000d3af9651a
  resourceVersion: "9547227"
  selfLink: /api/v1/namespaces/development/pods/httpd-test3-1-4zb4s
  uid: b42dc3ac-4826-11e8-990d-000d3af9651a
spec:
  containers:
  - image: docker-registry.default.svc:5000/development/httpd-test3@sha256:eea81bb9247dce67ded6fcacdeee10b310564b3e496089a68e65f61ad83a42e7
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    name: httpd-example
    ports:
    - containerPort: 8080
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    resources:
      limits:
        memory: 512Mi
      requests:
        memory: 512Mi
    securityContext:
      capabilities:
        drop:
        - MKNOD
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data3
      name: volume-wfrfs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-x8d24
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: default-dockercfg-r2gqq
  nodeName: ose3-single-vm
  nodeSelector:
    node-role.kubernetes.io/compute: "true"
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    seLinuxOptions:
      level: s0:c10,c5
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - name: volume-wfrfs
    persistentVolumeClaim:
      claimName: pvc-test2
  - name: default-token-x8d24
    secret:
      defaultMode: 420
      secretName: default-token-x8d24
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-04-25T01:19:30Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-04-25T01:20:46Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-04-25T01:19:30Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://d862001ed28f2b2a390b1095eede6322cdcd060a72cd822fad323fa090e33967
    image: docker-registry.default.svc:5000/development/httpd-test3:latest
    imageID: docker-pullable://docker-registry.default.svc:5000/development/httpd-test3@sha256:eea81bb9247dce67ded6fcacdeee10b310564b3e496089a68e65f61ad83a42e7
    lastState: {}
    name: httpd-example
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-04-25T01:20:41Z
  hostIP: 10.0.0.4
  phase: Running
  podIP: 10.130.0.67
  qosClass: Burstable
  startTime: 2018-04-25T01:19:30Z

Comment 10 Takayoshi Tanaka 2018-04-25 01:43:46 UTC
I understand the issue. However, we faced this issue after updating to the OCP 3.9. Before upgrading, the pod has a write permission. Do you know if there is any related code change?
I'd like to write a knowledgebase.

~~~
# oc get scc anyuid -o yaml
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities: null
allowedFlexVolumes: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
groups:
- system:authenticated
- system:cluster-admins
kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: anyuid provides all features of the restricted SCC
      but allows users to run with any UID and any GID.
  creationTimestamp: 2017-10-17T11:38:12Z
  name: anyuid
  resourceVersion: "7374906"
  selfLink: /apis/security.openshift.io/v1/securitycontextconstraints/anyuid
  uid: a87372a0-b32f-11e7-b1f6-000d3af9651a
priority: 10
readOnlyRootFilesystem: false
requiredDropCapabilities:
- MKNOD
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:default:default
- system:serviceaccount:fukuokanetconf:default
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret

Comment 11 Hemant Kumar 2018-04-25 01:56:51 UTC
Yep that is as well documented in the BZ I linked above. Something/someone has edited anyuid SCC and added `system:authenticated` to it. Also linked BZ has  important information about how this is supposed to work.

Comment 12 Takayoshi Tanaka 2018-04-25 02:15:17 UTC
Thanks Hemant,

I'll write a knowledgebase for customers to find this issue more easily.


Note You need to log in before you can comment on or make changes to this bug.