Bug 1277092 - Owner can't get container information via downward-api volume plugin because of "Permission denied"
Owner can't get container information via downward-api volume plugin because ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
3.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Paul Morie
chaoyang
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-02 05:35 EST by Qixuan Wang
Modified: 2016-05-12 13:09 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 13:09:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Qixuan Wang 2015-11-02 05:35:13 EST
Description of problem:
I want to enter a container as root, so I create a privileged pod with downward API volume plug-in, the privileged containers are not allowed to be created. So I have to set "privileged" as "false", the container could run, but container information isn't able to get because the owner can't read the file /etc/..2015_11_02_15_53_43215191314 although it has the permission. Plus, privileged pod without downward-api volume can be created.

Version-Release number of selected component (if applicable):
openshift v3.0.2.903-114-g2849767
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4
etcd 2.1.2


How reproducible:
Always

Steps to Reproduce:
1. Create a privileged pod that consumes its labele/annotations via the downward API volume.

# vi pod-dapi-volume.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-dapi-volume
  labels:
    region: r1
    zone: z11
    rack: a111
  annotations:
    build: one
    builder: qe-one
spec:
  containers:
    - name: client-container
      image: gcr.io/google_containers/busybox
      command: ["sh", "-c", "while true; do if [[ -e /etc/labels ]]; then cat /etc/labels; fi; if [[ -e /etc/annotations ]]; then cat /etc/annotations; fi; sleep 5; done"]
      securityContext:
        privileged: true
      volumeMounts:
        - name: podinfo
          mountPath: /etc
          readOnly: false
  volumes:
    - name: podinfo
      downwardAPI:
        items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations
          - path: "name"
            fieldRef:
              fieldPath: metadata.name
          - path: "namespace"
            fieldRef:
              fieldPath: metadata.namespace

# oc create -f pod-dapi-volume.yaml

2. Set "privileged" as "false" and then create the pod.
3. Enter into the container and check container info
# oc exec -it $pod_name sh
# ls -laR /etc


Actual results:
1. [root@dhcp-136-118 testv3]# oc create -f pod-dapi-volume.yaml
Error from server: error when creating "pod-dapi-volume.yaml": Pod "pod-dapi-volume-new" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.containers[0].securityContext.privileged: invalid value 'true', Details: Privileged containers are not allowed]


3. [root@dhcp-136-118 testv3]# oc exec -it pod-dapi-volume-new sh
/ $ id
uid=1000340000 gid=0 groups=1000340000
/ $ ls -laR /etc
/etc:
total 16
drwxrwxrwt    3 0        0              220 Nov  2 07:53 .
drwxr-xr-x   17 0        0             4096 Nov  2 07:53 ..
drwx------    2 0        0              120 Nov  2 07:53 ..2015_11_02_15_53_43215191314
lrwxrwxrwx    1 0        0               30 Nov  2 07:53 ..downwardapi -> ..2015_11_02_15_53_43215191314
lrwxrwxrwx    1 0        0               25 Nov  2 07:53 annotations -> ..downwardapi/annotations
-rw-r--r--    1 0        0               20 Nov  2 07:53 hostname
-rw-r--r--    1 0        0             1871 Nov  2 07:53 hosts
lrwxrwxrwx    1 0        0               20 Nov  2 07:53 labels -> ..downwardapi/labels
lrwxrwxrwx    1 0        0               18 Nov  2 07:53 name -> ..downwardapi/name
lrwxrwxrwx    1 0        0               23 Nov  2 07:53 namespace -> ..downwardapi/namespace
-rw-r--r--    1 0        0              166 Nov  2 07:53 resolv.conf

/etc/..2015_11_02_15_53_43215191314:
ls: can't open '/etc/..2015_11_02_15_53_43215191314': Permission denied
total 0


Expected results:
[root@ip-172-18-3-115 work]# oc exec -it pod-dapi-volume sh
/ # ls -laR /etc
/etc:
total 16
drwxrwxrwt    3 0        0              220 Oct 21 11:28 .
drwxr-xr-x   17 0        0             4096 Oct 21 11:31 ..
drwx------    2 0        0              120 Oct 21 11:28 ..2015_10_21_11_28_57085975213
lrwxrwxrwx    1 0        0               30 Oct 21 11:28 ..downwardapi -> ..2015_10_21_11_28_57085975213
lrwxrwxrwx    1 0        0               25 Oct 21 11:28 annotations -> ..downwardapi/annotations
-rw-r--r--    1 0        0               16 Oct 21 11:28 hostname
-rw-r--r--    1 0        0             2913 Oct 21 11:28 hosts
lrwxrwxrwx    1 0        0               20 Oct 21 11:28 labels -> ..downwardapi/labels
lrwxrwxrwx    1 0        0               18 Oct 21 11:28 name -> ..downwardapi/name
lrwxrwxrwx    1 0        0               23 Oct 21 11:28 namespace -> ..downwardapi/namespace
-rw-r--r--    1 0        0              139 Oct 21 11:28 resolv.conf

/etc/..2015_10_21_11_28_57085975213:
total 16
drwx------    2 0        0              120 Oct 21 11:28 .
drwxrwxrwt    3 0        0              220 Oct 21 11:28 ..
-rw-r--r--    1 0        0              152 Oct 21 11:28 annotations
-rw-r--r--    1 0        0               35 Oct 21 11:28 labels
-rw-r--r--    1 0        0               15 Oct 21 11:28 name
-rw-r--r--    1 0        0                6 Oct 21 11:28 namespace

Additional info:
Comment 1 Jordan Liggitt 2015-11-02 09:18:18 EST
To create privileged pods, the creating user must be added to the privileged security context constraints object:

oadm policy add-scc-to-user privileged your_username
Comment 2 weiwei jiang 2015-11-02 21:40:18 EST
(In reply to Jordan Liggitt from comment #1)
> To create privileged pods, the creating user must be added to the privileged
> security context constraints object:
> 
> oadm policy add-scc-to-user privileged your_username

Yeah, `oadm policy add-scc-to-user privileged your_username` really work. 
But downloadapi volume can work if and only if privileged=true?
And since downloadapi volume is created by origin, so 700 is expected?
Comment 3 Qixuan Wang 2015-11-02 22:44:26 EST
Yesterday we added user via "oc edit scc" under privileged section and found the problem. I think "oadm policy add-scc-to-user privileged your_username" plays the same role. Today I couldn't reproduce the problem although the new environment has the same version. It's weird.
Comment 4 weiwei jiang 2015-11-03 00:40:53 EST
(In reply to Jordan Liggitt from comment #1)
> To create privileged pods, the creating user must be added to the privileged
> security context constraints object:
> 
> oadm policy add-scc-to-user privileged your_username

And looks like the directory is created with 777, but do not know why it is 700.

https://github.com/openshift/origin/blob/master/Godeps/_workspace/src/k8s.io/kubernetes/pkg/volume/downwardapi/downwardapi.go#L285
Comment 5 Jordan Liggitt 2015-11-03 02:04:57 EST
Paul, any insight on the gid or downward API usage without making the container privileged?
Comment 6 Jordan Liggitt 2015-11-03 02:09:18 EST
Looks related to BZ1276038
Comment 7 Paul Weil 2015-11-03 08:38:05 EST
The downward api plugin is a read only volume.  Read only volumes were not being altered by the ownership and selinux management code.  That is being refactored to provide a better decision plane for read only volumes that may be managed by the kubelet.  Upstream PR is https://github.com/kubernetes/kubernetes/pull/16614.
Comment 8 Paul Morie 2016-01-05 09:10:05 EST
Should be fixed now.
Comment 9 Qixuan Wang 2016-01-05 22:19:02 EST
Need rebase to origin to verify it.
Comment 10 Qixuan Wang 2016-01-13 03:24:50 EST
Checked today's build devenv-fedora_3134(openshift v1.1-764-g7e08c95, kubernetes v1.1.0-origin-1107-g4c8e6f4, etcd 2.1.2), the upstream code has not been rebased into origin, so moved the bug to ASSIGNED temporarily.
Comment 11 Paul Morie 2016-02-03 17:15:15 EST
Sorry about that, should be fixed now that the rebase has landed.
Comment 12 Qixuan Wang 2016-02-05 02:14:27 EST
Tested on devevn-rhel_3335(openshift v1.1.1-440-g79b4613, kubernetes v1.2.0-alpha.4-851-g4a65fa1, etcd 2.2.2) which the upstream PR#16614 has been in it. 

Come into a privileged container as root so there is not permission problem. But enter into a non-privilege container of as a non-root role, the permission problem is still there. Steps are listed bellow. Could you help me to confirm we are in the same page? Thanks.

[root@ip-172-18-12-137 ~]# oc get pod
NAME                             READY     STATUS    RESTARTS   AGE
pod-dapi-volume-not-privileged   1/1       Running   0          1m
pod-dapi-volume-privileged       1/1       Running   0          2s

[root@ip-172-18-12-137 ~]# oc exec -it  pod-dapi-volume-not-privileged sh
/ $ ls -laR /etc
/etc:
total 16
drwxrwxrwt    3 0        0              220 Feb  5 06:58 .
drwxr-xr-x   17 0        0             4096 Feb  5 06:58 ..
drwx------    2 0        0              120 Feb  5 06:58 ..2016_02_05_01_58_19604757689
lrwxrwxrwx    1 0        0               30 Feb  5 06:58 ..downwardapi -> ..2016_02_05_01_58_19604757689
lrwxrwxrwx    1 0        0               25 Feb  5 06:58 annotations -> ..downwardapi/annotations
-rw-r--r--    1 0        0               31 Feb  5 06:58 hostname
-rw-r--r--    1 0        0              225 Feb  5 06:58 hosts
lrwxrwxrwx    1 0        0               20 Feb  5 06:58 labels -> ..downwardapi/labels
lrwxrwxrwx    1 0        0               18 Feb  5 06:58 name -> ..downwardapi/name
lrwxrwxrwx    1 0        0               23 Feb  5 06:58 namespace -> ..downwardapi/namespace
-rw-r--r--    1 0        0              140 Feb  5 06:58 resolv.conf

/etc/..2016_02_05_01_58_19604757689:
ls: can't open '/etc/..2016_02_05_01_58_19604757689': Permission denied
total 0



[root@ip-172-18-12-137 ~]# oc exec -it pod-dapi-volume-privileged sh
/ # ls -laR /etc
/etc:
total 16
drwxrwxrwt    3 0        0              220 Feb  5 06:59 .
drwxr-xr-x   17 0        0             4096 Feb  5 07:01 ..
drwx------    2 0        0              120 Feb  5 06:59 ..2016_02_05_01_59_29346186948
lrwxrwxrwx    1 0        0               30 Feb  5 06:59 ..downwardapi -> ..2016_02_05_01_59_29346186948
lrwxrwxrwx    1 0        0               25 Feb  5 06:59 annotations -> ..downwardapi/annotations
-rw-r--r--    1 0        0               27 Feb  5 06:59 hostname
-rw-r--r--    1 0        0              221 Feb  5 06:59 hosts
lrwxrwxrwx    1 0        0               20 Feb  5 06:59 labels -> ..downwardapi/labels
lrwxrwxrwx    1 0        0               18 Feb  5 06:59 name -> ..downwardapi/name
lrwxrwxrwx    1 0        0               23 Feb  5 06:59 namespace -> ..downwardapi/namespace
-rw-r--r--    1 0        0              140 Feb  5 06:59 resolv.conf

/etc/..2016_02_05_01_59_29346186948:
total 16
drwx------    2 0        0              120 Feb  5 06:59 .
drwxrwxrwt    3 0        0              220 Feb  5 06:59 ..
-rw-r--r--    1 0        0              157 Feb  5 06:59 annotations
-rw-r--r--    1 0        0               35 Feb  5 06:59 labels
-rw-r--r--    1 0        0               26 Feb  5 06:59 name
-rw-r--r--    1 0        0                6 Feb  5 06:59 namespace
Comment 13 Paul Morie 2016-02-08 11:38:26 EST
You need to have fsGroup set in order for this to work with a non-root UID.  You will need to use the 'anyuid' SCC or 'privileged' SCC, and put the fsGroup into the pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  securityContext:
    fsGroup: 1234
Comment 14 Qixuan Wang 2016-02-14 02:00:04 EST
Tested on devenv-rhel_3422(openshift v1.1.2-258-gda69009, kubernetes v1.2.0-origin, etcd 2.2.2+git), the bug has been fixed, thanks for your help.

Here are verification steps.
# cat pod-dapi-volume.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-dapi-volume
  labels:
    region: r1
    zone: z11
    rack: a111
  annotations:
    build: one
    builder: qe-one
spec:
  securityContext:
    fsGroup: 1234
  containers:
    - name: client-container
      image: gcr.io/google_containers/busybox
      command: ["sh", "-c", "while true; do if [[ -e /etc/labels ]]; then cat /etc/labels; fi; if [[ -e /etc/annotations ]]; then cat /etc/annotations; fi; sleep 5; done"]
      securityContext:
        privileged: false
      volumeMounts:
        - name: podinfo
          mountPath: /etc
          readOnly: false
  volumes:
    - name: podinfo
      downwardAPI:
        items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations
          - path: "name"
            fieldRef:
              fieldPath: metadata.name
          - path: "namespace"
            fieldRef:
              fieldPath: metadata.namespace

[root@ip-172-18-1-152 ~]# oc exec -it pod-dapi-volume sh
/ $ ls -laR /etc
/etc:
total 16
drwxrwsrwt    3 0        1234           220 Feb 14 06:53 .
drwxr-xr-x   17 0        0             4096 Feb 14 06:53 ..
drwxrwsrwx    2 0        1234           120 Feb 14 06:53 ..2016_02_14_01_53_09964771296
lrwxrwxrwx    1 0        0               30 Feb 14 06:53 ..downwardapi -> ..2016_02_14_01_53_09964771296
lrwxrwxrwx    1 0        0               25 Feb 14 06:53 annotations -> ..downwardapi/annotations
-rw-r--r--    1 0        0               16 Feb 14 06:53 hostname
-rw-r--r--    1 0        0              210 Feb 14 06:53 hosts
lrwxrwxrwx    1 0        0               20 Feb 14 06:53 labels -> ..downwardapi/labels
lrwxrwxrwx    1 0        0               18 Feb 14 06:53 name -> ..downwardapi/name
lrwxrwxrwx    1 0        0               23 Feb 14 06:53 namespace -> ..downwardapi/namespace
-rw-r--r--    1 0        0              139 Feb 14 06:53 resolv.conf

/etc/..2016_02_14_01_53_09964771296:
total 16
drwxrwsrwx    2 0        1234           120 Feb 14 06:53 .
drwxrwsrwt    3 0        1234           220 Feb 14 06:53 ..
-rwxrwsrwx    1 0        1234           157 Feb 14 06:53 annotations
-rwxrwsrwx    1 0        1234            35 Feb 14 06:53 labels
-rwxrwsrwx    1 0        1234            15 Feb 14 06:53 name
-rwxrwsrwx    1 0        1234             6 Feb 14 06:53 namespace

Note You need to log in before you can comment on or make changes to this bug.