Description of problem: I attach 2 secrets to a pod and try to mount one secret to /var/lib/base and the other secret to /var/lib/base/nested/. The nested secret does not show up as mounted to the pods. Version-Release number of selected component (if applicable): > openshift version openshift v3.6.173.0.49 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 How reproducible: 100% Steps to Reproduce: # mkdir ./basedir; echo "basefile1" > ./basedir/basefile1 ; echo "basefile2" > ./basedir/basefile2 # mkdir ./nestdir; echo "nestfile1" > ./nestdir/nestfile1 ; echo "nestfile2" > ./nestdir/nestfile2 # oc create secret generic basedir --from-file ./basedir # oc create secret generic nestdir --from-file ./nestdir # oc new-app --template=httpd-example # oc volume dc/httpd-example --add --mount-path=/var/lib/basedir -t secret --secret-name='basedir' --name basedir # oc volume dc/httpd-example --add --mount-path=/var/lib/basedir/nestdir -t secret --secret-name='nestdir' --name nestdir -- Pod shows as running and healthy Actual results: nested secret does not get mounted in pod. Expected results: secret to get mounted Additional info: - Check Docker and Node for mounts # mount | grep -e nestdir -e basedir tmpfs on /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir type tmpfs (rw,relatime,seclabel) tmpfs on /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir type tmpfs (rw,relatime,seclabel) # docker inspect 83878ec9ac1a | grep -e nestdir -e basedir "HostConfig": { "Binds": [ "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir:/var/lib/basedir:Z", "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir:/var/lib/basedir/nestdir:Z", ... ... "Mounts": [ { "Source": "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir", "Destination": "/var/lib/basedir", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, { "Source": "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir", "Destination": "/var/lib/basedir/nestdir", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, - Check in the pod from the node for mounts # nsenter -m -p -t $(docker inspect --format "{{ .State.Pid }}" 83878ec9ac1a) << NSEOF > mount | grep -e nestdir -e basedir > ls /var/lib/basedir/ -R > NSEOF tmpfs on /var/lib/basedir type tmpfs (rw,relatime,seclabel) /var/lib/basedir/: basefile1 basefile2 - Logs from node and docker Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.738478 8740 secret.go:186] Setting up volume basedir for pod 61189515-cfd8-11e7-ad20-fa163ec5d77e at /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.738872 8740 secret.go:186] Setting up volume nestdir for pod 61189515-cfd8-11e7-ad20-fa163ec5d77e at /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.740926 8740 secret.go:217] Received secret nesttest/basedir containing (2) pieces of data, 20 total bytes Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741029 8740 atomic_writer.go:333] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: current paths: [basefile1 basefile2] Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741048 8740 atomic_writer.go:345] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: new paths: [basefile1 basefile2] Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741059 8740 atomic_writer.go:348] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: paths to remove: map[] Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741147 8740 atomic_writer.go:142] pod nesttest/httpd-example-3-b2g8r volume basedir: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741286 8740 operation_generator.go:613] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/61189515-cfd8-11e7-ad20-fa163ec5d77e-basedir" (spec.Name: "basedir") pod "61189515-cfd8-11e7-ad20-fa163ec5d77e" (UID: "61189515-cfd8-11e7-ad20-fa163ec5d77e"). Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741799 8740 secret.go:217] Received secret nesttest/nestdir containing (2) pieces of data, 20 total bytes Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741875 8740 atomic_writer.go:333] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: current paths: [nestfile1 nestfile2] Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741884 8740 atomic_writer.go:345] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: new paths: [nestfile1 nestfile2] Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741890 8740 atomic_writer.go:348] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: paths to remove: map[] Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741925 8740 atomic_writer.go:142] pod nesttest/httpd-example-3-b2g8r volume nestdir: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.742073 8740 operation_generator.go:613] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/61189515-cfd8-11e7-ad20-fa163ec5d77e-nestdir" (spec.Name: "nestdir") pod "61189515-cfd8-11e7-ad20-fa163ec5d77e" (UID: "61189515-cfd8-11e7-ad20-fa163ec5d77e").
Issue is seen when booting wit kernel 3.10.0-693.5.2.el7.x86_64 After rolling kernel back to: 3.10.0-514.21.1.el7.x86_64 the issue is not seen and secrets are able to be mounted inside other secrets. Version of OpenShift did not change.
*** Bug 1516887 has been marked as a duplicate of this bug. ***
I have filed an issue upstream in the Kubernetes project: https://github.com/kubernetes/kubernetes/issues/57421 and I have also posted a pull request of code changes to address the issue: https://github.com/kubernetes/kubernetes/pull/57422 It's going to take some time for the community to review the pull request and make a decision about how to handle things. Assuming they like the patch as-is (or with minor modifications), we might see it merged early next year. Red Hat still hasn't made any decisions about which old versions of OpenShift should receive the patch, but once we do, I'd then backport the patch to those versions.
Merged in Origin: https://github.com/openshift/origin/pull/18165
Checked with # openshift version openshift v3.9.0-0.38.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8 and # uname -r 3.10.0-843.el7.x86_64 Can not reproduce this issue, so verify this.
Thanks for reporting this bug. My investigation of this bug led to the discovery of CVE-2017-1002102 (#1551818). Because the fixes for the security flaw also address this bug, you should be able to install an updated version to remedy this bug. I'm sure the errata tool will update with this bug with the information at some point, but in the mean time, you can get updated versions for each affected branch (the bug goes back to OpenShift 3.3): 3.3.1.46.11-1.git.4.e236015 3.4.1.44.38-1.git.4.bb8df08 3.5.5.31.48-1.git.4.ff6153e 3.6.173.0.96-1.git.4.e6301f8 3.7.23-1.git.5.83efd71
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489