Description of problem: When trigger MemoryPressure, the eviction manager can't find any starved resource, I wait more than 30 minutes, but node still keep in MemoryPressure=True; After 30 minutes, I restart the node service, the node become MemoryPressure=False Version-Release number of selected component (if applicable): openshift v3.5.0.17+c55cf2b kubernetes v1.5.2+43a9be4 etcd 3.1.0 instance on ec2: m3.large How reproducible: Sometime Steps to Reproduce: 1. Configure node soft memory eviction and restart node service kubeletArguments: eviction-soft: - "memory.available<3.5Gi" eviction-soft-grace-period: - "memory.available=30s" eviction-max-pod-grace-period: - "10" eviction-pressure-transition-period: - "1m0s" 2. Create some pod trigger memory pressure (note: new created pod in the same node and others pods located in this node before) for i in {1..5}; do kubectl create -f https://raw.githubusercontent.com/derekwaynecarr/kubernetes/examples-eviction/demo/kubelet-eviction/besteffort-pod.yaml -n dma ; done for i in {1..5}; do kubectl create -f https://raw.githubusercontent.com/derekwaynecarr/kubernetes/examples-eviction/demo/kubelet-eviction/burstable-pod.yaml -n dma ; done for i in {1..5}; do kubectl create -f https://raw.githubusercontent.com/derekwaynecarr/kubernetes/examples-eviction/demo/kubelet-eviction/guaranteed-pod.yaml -n dma ; done 3. Check node status and node logs Actual results: 3. Node keep in MemoryPressure and can't resume, eviction manager can't find any pod to evict //logs in node Feb 7 04:40:53 ip-172-18-6-5 atomic-openshift-node: I0207 04:40:53.646773 29576 helpers.go:734] eviction manager: eviction criteria not yet met for threshold(signal=memory.available, operator=3584Mi, value=LessThan, gracePeriod=30s), duration: 0s Feb 7 04:40:53 ip-172-18-6-5 atomic-openshift-node: I0207 04:40:53.646819 29576 eviction_manager.go:269] eviction manager: no resources are starved Expected results: Additional info:
I will try and reproduce.
I attempted to reproduce this issue. In my reproduction, memory pressure was raised, pods were evicted, and after ~1 min, the memory pressure was returned back to false. With the 3.5Gi eviction threshold, the kubelet just evicts pods, but that may not result in a full reclaim above the specified threshold depending on other conditions on the node. The kubelet restarting may have actually enabled reclaim by the kernel for additional memory. I am going to add additional logging to the eviction manager so we can capture better logging output in service of this issue, but I suspect if we ran the following: $ while true; do oc get --raw api/v1/nodes/<nodename>/proxy/stats/summary > `date "+%Y.%m.%d-%H.%M.%S"`-summary.log; sleep 10; done We would have seen memory.available was still causing the threshold to be met. Can we confirm that memory.available was actually above the specified value on the node, and that the kubelet still did not update the corresponding node condition?
I have opened a new PR upstream to improve logging in eviction manager so we can understand the state of the system better. It will output what the kubelet observes for each signal, and what thresholds or node conditions are met w/ their associated grace periods. This will help us understand how in some cases the node is still not able to return above a specified threshold with pod eviction alone. UPSTREAM PR: https://github.com/kubernetes/kubernetes/pull/41147
ORIGIN PR: https://github.com/openshift/origin/pull/12876 I am moving this bug to POST so we can evaluate in a subsequent validation that the node was acting properly because eviction alone did not provide sufficient memory reclaim with the new logs.
This has been merged into ocp and is in OCP v3.5.0.19 or newer.
Verify the bug on openshift v3.5.0.20+87266c6, when this happen again, I'll reopen the bug. [root@openshift-104 dma]# oc describe no openshift-104.lab.sjc.redhat.com Name: openshift-104.lab.sjc.redhat.com Role: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/hostname=openshift-104.lab.sjc.redhat.com role=node Taints: <none> CreationTimestamp: Wed, 15 Feb 2017 00:24:33 -0500 Phase: Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Wed, 15 Feb 2017 02:48:31 -0500 Wed, 15 Feb 2017 00:24:33 -0500 KubeletHasSufficientDisk kubelet has sufficient disk space available MemoryPressure False Wed, 15 Feb 2017 02:48:31 -0500 Wed, 15 Feb 2017 02:47:41 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 15 Feb 2017 02:48:31 -0500 Wed, 15 Feb 2017 00:24:33 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure Ready True Wed, 15 Feb 2017 02:48:31 -0500 Wed, 15 Feb 2017 02:36:31 -0500 KubeletReady kubelet is posting ready status Addresses: 10.14.6.104,10.14.6.104,openshift-104.lab.sjc.redhat.com Capacity: alpha.kubernetes.io/nvidia-gpu: 0 cpu: 2 memory: 3881932Ki pods: 250 Allocatable: alpha.kubernetes.io/nvidia-gpu: 0 cpu: 2 memory: 3881932Ki pods: 250 System Info: Machine ID: acefceb9946d4bc59cee8c921f22116f System UUID: 690F4796-19EA-4E60-B8AD-E4C817AA9623 Boot ID: e1930b45-242b-486e-9710-228904137335 Kernel Version: 3.10.0-514.10.1.el7.x86_64 OS Image: Red Hat Enterprise Linux Server 7.3 (Maipo) Operating System: linux Architecture: amd64 Container Runtime Version: docker://1.12.6 Kubelet Version: v1.5.2+43a9be4 Kube-Proxy Version: v1.5.2+43a9be4 ExternalID: openshift-104.lab.sjc.redhat.com Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits --------- ---- ------------ ---------- --------------- ------------- dma guaranteed-dgwp8 100m (5%) 100m (5%) 300Mi (7%) 300Mi (7%) dma guaranteed-fbcrc 100m (5%) 100m (5%) 300Mi (7%) 300Mi (7%) dma guaranteed-j6l1z 100m (5%) 100m (5%) 300Mi (7%) 300Mi (7%) dma guaranteed-kwhq7 100m (5%) 100m (5%) 300Mi (7%) 300Mi (7%) dma guaranteed-rfqmx 100m (5%) 100m (5%) 300Mi (7%) 300Mi (7%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted. CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 500m (25%) 500m (25%) 1500Mi (39%) 1500Mi (39%) Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 2h 31m 3 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeNotSchedulable Node openshift-104.lab.sjc.redhat.com status is now: NodeNotSchedulable 37m 26m 3 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeSchedulable Node openshift-104.lab.sjc.redhat.com status is now: NodeSchedulable 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal Starting Starting kubelet. 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Warning ImageGCFailed unable to find data for container / 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasSufficientDisk Node openshift-104.lab.sjc.redhat.com status is now: NodeHasSufficientDisk 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasSufficientMemory Node openshift-104.lab.sjc.redhat.com status is now: NodeHasSufficientMemory 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasNoDiskPressure Node openshift-104.lab.sjc.redhat.com status is now: NodeHasNoDiskPressure 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeNotReady Node openshift-104.lab.sjc.redhat.com status is now: NodeNotReady 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasInsufficientMemory Node openshift-104.lab.sjc.redhat.com status is now: NodeHasInsufficientMemory 24m 24m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeReady Node openshift-104.lab.sjc.redhat.com status is now: NodeReady 16m 12m 12 {kubelet openshift-104.lab.sjc.redhat.com} Warning EvictionThresholdMet Attempting to reclaim memory 12m 12m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal Starting Starting kubelet. 12m 12m 1 {kubelet openshift-104.lab.sjc.redhat.com} Warning ImageGCFailed unable to find data for container / 12m 12m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasSufficientDisk Node openshift-104.lab.sjc.redhat.com status is now: NodeHasSufficientDisk 12m 12m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasNoDiskPressure Node openshift-104.lab.sjc.redhat.com status is now: NodeHasNoDiskPressure 12m 12m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeNotReady Node openshift-104.lab.sjc.redhat.com status is now: NodeNotReady 12m 12m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeReady Node openshift-104.lab.sjc.redhat.com status is now: NodeReady 5m 5m 1 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasInsufficientMemory Node openshift-104.lab.sjc.redhat.com status is now: NodeHasInsufficientMemory 5m 4m 7 {kubelet openshift-104.lab.sjc.redhat.com} Warning SystemOOM System OOM encountered 5m 2m 7 {kubelet openshift-104.lab.sjc.redhat.com} Warning EvictionThresholdMet Attempting to reclaim memory 12m 57s 3 {kubelet openshift-104.lab.sjc.redhat.com} Normal NodeHasSufficientMemory Node openshift-104.lab.sjc.redhat.com status is now: NodeHasSufficientMemory
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884