Bug 1276038

Reassigning to Storage since this isn't a deployment issue.

What is the selinux label of your node's volume dir?

I don't think it's a selinux issue, just raw permissions/ownership at play here: the data is put into a directory with a timestamp (symlinked from .current) which is owned by root, mode 0700.

Some more data just in case: looking at the volume from the node:

[root@ose-node-compute-2d795 ~]# mount | grep podinfo
tmpfs on /var/lib/openshift/openshift.local.volumes/pods/87deaeee-7d77-11e5-a033-525400d9bd5d/volumes/kubernetes.io~metadata/podinf  type tmpfs (rw,relatime,rootcontext=system_u:object_r:svirt_sandbox_file_t:s0,seclabel)

[root@ose-node-compute-2d795 ~]# cd /var/lib/openshift/openshift.local.volumes/pods/87deaeee-7d77-11e5-a033-525400d9bd5d/volumes/kubernetes.io~metadata/podinfo/
[root@ose-node-compute-2d795 podinfo]# ls -laZ
drwxrwxrwt. root root system_u:object_r:svirt_sandbox_file_t:s0 .
drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0 ..
drwx------. root root system_u:object_r:svirt_sandbox_file_t:s0 .2015_10_29_05_56_06909120942
lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0 annotations -> .current/annotations
lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0 .current -> .2015_10_29_05_56_06909120942
lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0 labels -> .current/labels

SELinux labels look correct, right?

Trying to access it from the pod:

$ oc rsh cakephp-example-2-n6bfr
bash-4.2$ cd /var/tmp/podinfo
bash-4.2$ ls -la
total 4
drwxrwxrwt. 3 root root  120 Oct 29 05:56 .
drwxrwxrwt. 3 root root 4096 Oct 29 05:56 ..
drwx------. 2 root root   80 Oct 29 05:56 .2015_10_29_05_56_06909120942
lrwxrwxrwx. 1 root root   29 Oct 29 05:56 .current -> .2015_10_29_05_56_06909120942
lrwxrwxrwx. 1 root root   20 Oct 29 05:56 annotations -> .current/annotations
lrwxrwxrwx. 1 root root   15 Oct 29 05:56 labels -> .current/labels
bash-4.2$ cat labels
cat: labels: Permission denied

The node did not record deinals about that:

[root@ose-node-compute-2d795 podinfo]# ausearch -m avc -ts recent
<no matches>

Proposed upstream fix: https://github.com/kubernetes/kubernetes/pull/16614

The root issue here is that some volumes are managed by the kubelet but are read-only to pods.  The linked PR adds a way to account for the distinction and appropriately manage ownership on these types of volumes.

Should be fixed now.

Need rebase to origin to verify it. So move the bug to MODIFIED temporarily.

Not a 3.1.1 blocker. It will be delivered with the next Kube rebase into Origin.

Should be in master now.

Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1277092

Tested with the latest puddle: atomic-openshift-3.1.1.901-1.git.0.1aee00d.el7.x86_64

# openshift version
openshift v3.1.1.901
kubernetes v1.2.0-origin
etcd 2.2.2+git

The problem has been fixed on origin but still here:

# oc rsh pod-dapi-volume
bash-4.2$ ls -laR /var/tmp/podinfo/
/var/tmp/podinfo/:
total 0
drwxrwxrwt. 3 root root 120 Feb 14 06:27 .
drwxrwxrwt. 3 root root  20 Feb 14 06:28 ..
drwx------. 2 root root  80 Feb 14 06:27 ..2016_02_14_19_27_45033058333
lrwxrwxrwx. 1 root root  30 Feb 14 06:27 ..downwardapi -> ..2016_02_14_19_27_45033058333
lrwxrwxrwx. 1 root root  25 Feb 14 06:27 annotations -> ..downwardapi/annotations
lrwxrwxrwx. 1 root root  20 Feb 14 06:27 labels -> ..downwardapi/labels
ls: cannot open directory /var/tmp/podinfo/..2016_02_14_19_27_45033058333: Permission denied

Here is the pod spec file.
# cat pod-dapi-volume.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-dapi-volume
  labels:
    region: r1
    zone: z11
    rack: a111
  annotations:
    build: one
    builder: qe-one
spec:
  securityContext:
    fsGroup: 1234
  containers:
    - name: client-container
      image: openshift3/ruby-20-rhel7
      command: ["sh", "-c", "while true; do if [[ -e /var/tmp/podinfo/labels ]]; then cat /var/tmp/podinfo/labels; fi; if [[ -e /var/tmp/podinfo/annotations ]]; then cat /var/tmp/podinfo/annotations; fi; sleep 5; done"] 
      securityContext:
        privileged: false
      volumeMounts:
        - name: podinfo
          mountPath: /var/tmp/podinfo
          readOnly: false
  volumes:
    - name: podinfo
      metadata:
        items:
          - name: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - name: "annotations"
            fieldRef:
              fieldPath: metadata.annotations

It looks like the code changes should have made it into this version.  I'm seeing if I can recreate now.

I ran the E2E test for this against a 3.1.1.901 cluster and it passed.  So, would you please reproduce this and include:

1.  openshift-node logs
2.  output of `oc get <pod name> -o yaml`, note, this will be different than the pod descriptor you submit
3.  output of `oc describe <pod name>`

That will help us track this down.  Thanks!

Changing needinfo for this, sorry Pep.

When you recreate, please use --loglevel=5

Created attachment 1127499 [details]
atomic-openshift-node logs

Tested with the latest puddle: atomic-openshift-3.1.1.902-1.git.0.d625c01.el7.x86_64

# openshift version
openshift v3.1.1.902
kubernetes v1.2.0-origin
etcd 2.2.2+git

Kept OPTIONS=--loglevel=5 on /etc/sysconfig/atomic-openshift-node

I noticed that "fsGroup" is not wrote into securityContext which is different with that against Origin. 

OSE/AEP:
securityContext: {}

Origin:
securityContext:
    fsGroup: 1234
    seLinuxOptions:
      level: s0:c6,c0


Node log is attached. Here are more infos.
# oc get pod pod-dapi-volume -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    build: one
    builder: qe-one
    openshift.io/scc: restricted
  creationTimestamp: 2016-02-16T07:21:59Z
  labels:
    rack: a111
    region: r1
    zone: z11
  name: pod-dapi-volume
  namespace: qwang1
  resourceVersion: "2138"
  selfLink: /api/v1/namespaces/qwang1/pods/pod-dapi-volume
  uid: f7bed65e-d47d-11e5-9ee2-fa163ee7299c
spec:
  containers:
  - command:
    - sh
    - -c
    - while true; do if [[ -e /var/tmp/podinfo/labels ]]; then cat /var/tmp/podinfo/labels;
      fi; if [[ -e /var/tmp/podinfo/annotations ]]; then cat /var/tmp/podinfo/annotations;
      fi; sleep 5; done
    image: openshift3/ruby-20-rhel7
    imagePullPolicy: IfNotPresent
    name: client-container
    resources: {}
    securityContext:
      capabilities:
        drop:
        - KILL
        - MKNOD
        - SETGID
        - SETUID
        - SYS_CHROOT
      privileged: false
      runAsUser: 1000040000
      seLinuxOptions:
        level: s0:c6,c5
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/tmp/podinfo
      name: podinfo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-0ufgv
      readOnly: true
  dnsPolicy: ClusterFirst
  host: openshift-148.lab.eng.nay.redhat.com
  imagePullSecrets:
  - name: default-dockercfg-ufek6
  nodeName: openshift-148.lab.eng.nay.redhat.com
  restartPolicy: Always
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - downwardAPI:
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        path: labels
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    metadata:
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        name: labels
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        name: annotations
    name: podinfo
  - name: default-token-0ufgv
    secret:
      secretName: default-token-0ufgv
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2016-02-16T07:22:03Z
    status: "True"
    type: Ready
  containerStatuses:
  - containerID: docker://b7d01010da9a9815928ef759bff0207acaf9205c36496ea2447b5093ed6d4b53
    image: openshift3/ruby-20-rhel7
    imageID: docker://be90266e447e20b841eb19f856dcc616a7cbf23ecdd9779d0206aa140b632031
    lastState: {}
    name: client-container
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2016-02-16T07:22:03Z
  hostIP: 10.66.79.148
  phase: Running
  podIP: 10.2.0.5
  startTime: 2016-02-16T07:21:59Z


# oc describe pod pod-dapi-volume
Name:				pod-dapi-volume
Namespace:			qwang1
Image(s):			openshift3/ruby-20-rhel7
Node:				openshift-148.lab.eng.nay.redhat.com/10.66.79.148
Start Time:			Tue, 16 Feb 2016 15:21:59 +0800
Labels:				rack=a111,region=r1,zone=z11
Status:				Running
Reason:				
Message:			
IP:				10.2.0.5
Replication Controllers:	<none>
Containers:
  client-container:
    Container ID:	docker://b7d01010da9a9815928ef759bff0207acaf9205c36496ea2447b5093ed6d4b53
    Image:		openshift3/ruby-20-rhel7
    Image ID:		docker://be90266e447e20b841eb19f856dcc616a7cbf23ecdd9779d0206aa140b632031
    QoS Tier:
      cpu:		BestEffort
      memory:		BestEffort
    State:		Running
      Started:		Tue, 16 Feb 2016 15:22:03 +0800
    Ready:		True
    Restart Count:	0
    Environment Variables:
Conditions:
  Type		Status
  Ready 	True 
Volumes:
  podinfo:
  <Volume Type Not Found>
  default-token-0ufgv:
    Type:	Secret (a secret that should populate this volume)
    SecretName:	default-token-0ufgv
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath				Reason		Message
  ─────────	────────	─────	────						─────────────				──────		───────
  57s		57s		1	{default-scheduler }									Scheduled	Successfully assigned pod-dapi-volume to openshift-148.lab.eng.nay.redhat.com
  55s		55s		1	{kubelet openshift-148.lab.eng.nay.redhat.com}	spec.containers{client-container}	Pulled		Container image "openshift3/ruby-20-rhel7" already present on machine
  54s		54s		1	{kubelet openshift-148.lab.eng.nay.redhat.com}	spec.containers{client-container}	Created		Created container with docker id b7d01010da9a
  53s		53s		1	{kubelet openshift-148.lab.eng.nay.redhat.com}	spec.containers{client-container}	Started		Started container with docker id b7d01010da9a

You need to have fsGroup in the pod's security context for this to work as a non-root UID.  Try running with the 'anyuid' SCC and setting fsGroup to 1234 manually.

Actually, disregard my comment re: anyuid SCC.  Specifying FSGroup manually should be enough.

Used pod yaml file which was specified fsGroup in Comment 13 and got results in Comment 19. The only reason is I used previous openshift locally due to my stupid script didn't check it's failed update. I apologize for my negligence. The bug has been fixed. Thanks again!

Is this fix part of the currently released Atomic Host

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064