Bug 1516569

Summary:	Unable to mount a secret inside another secrets mount point with kernel 3.10.0-693.5.2
Product:	OpenShift Container Platform	Reporter:	Ryan Howe <rhowe>
Component:	Node	Assignee:	Joel Smith <joelsmith>
Status:	CLOSED ERRATA	QA Contact:	DeShuai Ma <dma>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	3.6.0	CC:	amurdaca, aos-bugs, dwalsh, joelsmith, jokerman, mmccomas, pdwyer, sjenning, sreber, vgoyal, wjiang
Target Milestone:	---	Keywords:	TestCaseNeeded
Target Release:	3.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-28 14:13:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1267746

Description Ryan Howe 2017-11-22 23:24:17 UTC

Description of problem:

I attach 2 secrets to a pod and try to mount one secret to /var/lib/base and the other secret to /var/lib/base/nested/. The nested secret does not show up as mounted to the pods. 


Version-Release number of selected component (if applicable):

> openshift version
openshift v3.6.173.0.49
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
100%


Steps to Reproduce:

# mkdir ./basedir; echo "basefile1" > ./basedir/basefile1 ; echo "basefile2" > ./basedir/basefile2
# mkdir ./nestdir;  echo "nestfile1" > ./nestdir/nestfile1 ; echo "nestfile2" > ./nestdir/nestfile2

# oc create secret generic basedir --from-file ./basedir
# oc create secret generic nestdir --from-file ./nestdir

# oc new-app --template=httpd-example
# oc volume dc/httpd-example  --add --mount-path=/var/lib/basedir -t secret --secret-name='basedir' --name basedir
# oc volume dc/httpd-example  --add --mount-path=/var/lib/basedir/nestdir -t secret --secret-name='nestdir' --name nestdir

-- Pod shows as running and healthy 

Actual results:
nested secret does not get mounted in pod. 


Expected results:
secret to get mounted

Additional info:


- Check Docker and Node for mounts 
# mount | grep -e nestdir -e basedir
tmpfs on /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir type tmpfs (rw,relatime,seclabel)
tmpfs on /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir type tmpfs (rw,relatime,seclabel)

# docker inspect 83878ec9ac1a |  grep -e nestdir -e basedir

        "HostConfig": {
            "Binds": [
                "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir:/var/lib/basedir:Z",
                "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir:/var/lib/basedir/nestdir:Z",
         ...
         ...
        "Mounts": [
            {
                "Source": "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir",
                "Destination": "/var/lib/basedir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir",
                "Destination": "/var/lib/basedir/nestdir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },


- Check in the pod from the node for mounts
# nsenter -m -p  -t $(docker inspect --format "{{ .State.Pid }}" 83878ec9ac1a) << NSEOF
> mount | grep -e nestdir -e basedir
> ls /var/lib/basedir/ -R
> NSEOF

tmpfs on /var/lib/basedir type tmpfs (rw,relatime,seclabel)
/var/lib/basedir/:
basefile1  basefile2



- Logs from node and docker
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.738478    8740 secret.go:186] Setting up volume basedir for pod 61189515-cfd8-11e7-ad20-fa163ec5d77e at /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.738872    8740 secret.go:186] Setting up volume nestdir for pod 61189515-cfd8-11e7-ad20-fa163ec5d77e at /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.740926    8740 secret.go:217] Received secret nesttest/basedir containing (2) pieces of data, 20 total bytes
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741029    8740 atomic_writer.go:333] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: current paths:   [basefile1 basefile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741048    8740 atomic_writer.go:345] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: new paths:       [basefile1 basefile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741059    8740 atomic_writer.go:348] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: paths to remove: map[]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741147    8740 atomic_writer.go:142] pod nesttest/httpd-example-3-b2g8r volume basedir: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741286    8740 operation_generator.go:613] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/61189515-cfd8-11e7-ad20-fa163ec5d77e-basedir" (spec.Name: "basedir") pod "61189515-cfd8-11e7-ad20-fa163ec5d77e" (UID: "61189515-cfd8-11e7-ad20-fa163ec5d77e").
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741799    8740 secret.go:217] Received secret nesttest/nestdir containing (2) pieces of data, 20 total bytes
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741875    8740 atomic_writer.go:333] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: current paths:   [nestfile1 nestfile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741884    8740 atomic_writer.go:345] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: new paths:       [nestfile1 nestfile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741890    8740 atomic_writer.go:348] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: paths to remove: map[]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741925    8740 atomic_writer.go:142] pod nesttest/httpd-example-3-b2g8r volume nestdir: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.742073    8740 operation_generator.go:613] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/61189515-cfd8-11e7-ad20-fa163ec5d77e-nestdir" (spec.Name: "nestdir") pod "61189515-cfd8-11e7-ad20-fa163ec5d77e" (UID: "61189515-cfd8-11e7-ad20-fa163ec5d77e").

Comment 1 Ryan Howe 2017-11-23 17:42:42 UTC

Issue is seen when booting wit kernel 3.10.0-693.5.2.el7.x86_64

After rolling kernel back to: 3.10.0-514.21.1.el7.x86_64 the issue is not seen and secrets are able to be mounted inside other secrets. 

Version of OpenShift did not change.

Comment 2 Paul Dwyer 2017-11-24 09:08:57 UTC

*** Bug 1516887 has been marked as a duplicate of this bug. ***

Comment 30 Joel Smith 2017-12-21 00:48:58 UTC

I have filed an issue upstream in the Kubernetes project:

https://github.com/kubernetes/kubernetes/issues/57421

and I have also posted a pull request of code changes to address the issue:

https://github.com/kubernetes/kubernetes/pull/57422

It's going to take some time for the community to review the pull request and make a decision about how to handle things. Assuming they like the patch as-is (or with minor modifications), we might see it merged early next year.

Red Hat still hasn't made any decisions about which old versions of OpenShift should receive the patch, but once we do, I'd then backport the patch to those versions.

Comment 31 Seth Jennings 2018-01-31 17:30:37 UTC

Merged in Origin:
https://github.com/openshift/origin/pull/18165

Comment 33 weiwei jiang 2018-02-07 09:10:21 UTC

Checked with 
# openshift version 
openshift v3.9.0-0.38.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8

and
# uname -r
3.10.0-843.el7.x86_64

Can not reproduce this issue, so verify this.

Comment 34 Joel Smith 2018-03-12 20:28:17 UTC

Thanks for reporting this bug. My investigation of this bug led to the discovery of CVE-2017-1002102 (#1551818).

Because the fixes for the security flaw also address this bug, you should be able to install an updated version to remedy this bug. I'm sure the errata tool will update with this bug with the information at some point, but in the mean time, you can get updated versions for each affected branch (the bug goes back to OpenShift 3.3):

3.3.1.46.11-1.git.4.e236015
3.4.1.44.38-1.git.4.bb8df08
3.5.5.31.48-1.git.4.ff6153e
3.6.173.0.96-1.git.4.e6301f8
3.7.23-1.git.5.83efd71

Comment 38 errata-xmlrpc 2018-03-28 14:13:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489