Bug 1627767 - mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied while using storage class kubernetes.io/scaleio
Summary: mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission den...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.6.z
Assignee: Matthew Wong
QA Contact: Wenqi He
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-11 12:20 UTC by Miheer Salunke
Modified: 2018-09-14 13:45 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-14 13:45:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 3 Ben Parees 2018-09-11 12:30:37 UTC
Based on:

Sep 11 15:22:46 pagent10 dockerd-current[2570]: mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

This is a storage permissions issue, not an image bug.

Either the /var/lib/pgsql/data volume has incorrect permissions, or the /var/lib/pgsql/data/userdata dir which was created previously, now has permissions that do not match the new container that is mounting it.

Comment 5 Miheer Salunke 2018-09-11 14:09:19 UTC
Performed the selinux changes but didn't work. Still getting the permission denied issues.

1.
Turn off selinux one node node1.example.com which you see by oc get nodes

then 

oc patch dc postgresql -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname": "node1.example.com"}}}}}

then the pod shall get scheduled on that node.

Manualdeployment->

oc scale dc <dcname> --replicas=0
oc scale dc <dc name> --replicas=1
 
or you can oc rollout latest <dc name> if deployment does not trigger.
 


2.
Regarding storage :

Storage Used: EMC ScaleIO 2.6
Type: Software-Defined
Communication: Agent Based

Comment 6 Matthew Wong 2018-09-13 14:30:56 UTC
We can try to create a test pod to read/write the same PVC postgresql. If the same error doesn't occur, the test pod can be used to check what permissions the backing scaleio PV has. e.g.
```
kind: Pod
apiVersion: v1
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
       claimName: postgresql
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage
```
and then check the permissions inside the pod on "/usr/share/nginx/html"

I am not sure about tomcat, but the postgres has a "USER 26" instruction in its Dockerfile. This should not be needed, but what if 
securityContext: 
    fsGroup: 26
is set in the deploymentConfig?

Please provide also the specs of the PV to which the PVC is bound (pvc-d709f58c-b59b-11e8-9043-005056956b03) and the StorageClass "database".

Thanks.


Note You need to log in before you can comment on or make changes to this bug.