Bug 1422049
| Summary: | EmptyDir could lead to memory exhaustion | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sergi Jimenez Romero <sjr> |
| Component: | Node | Assignee: | Seth Jennings <sjenning> |
| Status: | CLOSED ERRATA | QA Contact: | Qixuan Wang <qixuan.wang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.3.0 | CC: | aos-bugs, decarr, fanlong_meng, jokerman, mmccomas, qixuan.wang, sjenning, sreber, wmeng |
| Target Milestone: | --- | ||
| Target Release: | 3.7.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: A design limitation in previous versions does not account memory-backed volumes against the pod's cumulative memory limit.
Consequence: It is possible for a user to exhaust memory on the node by creating a large file in an memory-backed volume, regardless of the memory limit.
Fix: Pod-level cgroups were added to, among other things, enforce limits on memory-backed volumes.
Result: Memory-backed volume sizes are now bounded by cumulative pod memory limits.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-28 21:52:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Sergi Jimenez Romero
2017-02-14 11:24:49 UTC
The 2 references: [1] https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt [2] https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt To address this issue, we need pod level cgroup hierarchy planned in Kubernetes 1.6. I just verified that having pod cgroups enabled on the node in 3.6 (it is enabled by default) enforces the memory limit wrt memory backed emptydirs.
[root@test ~]# cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: busybox
resources:
limits:
memory: 1Gi
cpu: 1
command:
- dd
- if=/dev/zero
- of=/mnt/zero
- bs=1M
- count=2000
volumeMounts:
- name: myvol
mountPath: /mnt
terminationGracePeriodSeconds: 0
volumes:
- name: myvol
emptyDir:
medium: Memory
[root@test ~]# oc create -f busybox.yaml
pod "busybox" created
[root@test ~]# oc describe pod | grep -A 5 "Last State"
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Fri, 05 May 2017 17:10:42 +0000
Finished: Fri, 05 May 2017 17:10:42 +0000
Ready: False
# mount | grep myvol
tmpfs on /var/lib/origin/openshift.local.volumes/pods/c296d664-31b5-11e7-a96c-fa163e71bc65/volumes/kubernetes.io~empty-dir/myvol type tmpfs (rw,relatime,seclabel)
[root@test ~]# cd /var/lib/origin/openshift.local.volumes/pods/c296d664-31b5-11e7-a96c-fa163e71bc65/volumes/kubernetes.io~empty-dir/myvol
[root@test myvol]# ls -alh
total 1023M
drwxrwsrwt. 2 root 1000040000 60 May 5 17:10 .
drwxr-xr-x. 3 root root 19 May 5 17:10 ..
-rw-r--r--. 1 1000040000 1000040000 1023M May 5 17:11 zero
Even though the pod tries to write a 2Gi file, it is OOMKilled when the file reached 1Gi in size i.e. the memory limit set on the container.
Upstream PR: https://github.com/kubernetes/kubernetes/pull/41349 Included in Origin 1.6.1 rebase: https://github.com/openshift/origin/pull/13653 Tested on OCP3.6 (openshift v3.6.79, kubernetes v1.6.1+5115d708d7, etcd 3.1.0) EmptyDir won't exhaust memory. Move the bug to VERIFIED, thanks. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188 you can use sizeLimit and i verified it already.
even though after entering container, df -h , you see the emptydir is 128G, but you could only use the space under the sizeLimit.
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: gcr.io/google_containers/busybox:1.24
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 1Gi
cpu: 1
command: ['sh', '-c', 'echo Hello Kubernetes!>/test-pd/mfltest.txt && sleep 3600' ]
ports:
- containerPort: 80
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
emptyDir:
medium: Memory
sizeLimit: "1M"
after enter the container, you can verify by typing: dd if=/dev/zero of=/test-pd/zero bs=1M count=10
the container exit.
if you have further question, please let met know fanlong_meng
|