Bug 1801826 - [4.2] A pod that gradually leaks memory causes node to become unreachable for 10 minutes
Summary: [4.2] A pod that gradually leaks memory causes node to become unreachable for...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.z
Assignee: Ryan Phillips
QA Contact: Sunil Choudhary
URL:
Whiteboard:
: 1771016 1776185 1797828 (view as bug list)
Depends On: 1800319 1801824 1802687 1806786 1808429 1808444 1810136
Blocks: OCP/Z_4.2 1776821 1801829
TreeView+ depends on / blocked
 
Reported: 2020-02-11 17:10 UTC by Ryan Phillips
Modified: 2020-04-20 16:10 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1801824
: 1801829 (view as bug list)
Environment:
Last Closed: 2020-02-25 21:45:58 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Ryan Phillips 2020-02-11 17:10:59 UTC
+++ This bug was initially created as a clone of Bug #1801824 +++

+++ This bug was initially created as a clone of Bug #1800319 +++

Creating a memory hogger pod (that should be evicted / OOM killed) instead of being safely handled by the node causes the node to become unreachable for >10m.  On the node, the kubelet appears to be running but can't heartbeat the apiserver.  Also, the node appears to think that the apiserver deleted all the pods (DELETE("api") in logs) which is not correct - no pods except the oomkilled one should be evicted / deleted.

Recreate

1. Create the attached kill-node.yaml on the cluster (oc create -f kill-node.yaml)
2. Wait 2-3 minutes while memory fills up on the worker

Expected:

1. memory-hog pod is oomkilled and/or evicted (either would be acceptable)
2. the node remains ready

Actual:

1. Node is tainted as unreachable, heartbeats stop, and it takes >10m for it to recover
2. After recovery, events are delivered

As part of fixing this, we need to add an e2e tests to the origin disruptive suite that triggers this (and add eviction tests, because this doesn't seem to evict anything).

--- Additional comment from Clayton Coleman on 2020-02-06 21:14:33 UTC ---

Once this is fixed we need to test against 4.3 and 4.2 and backport if it happens - this can DoS a node.

Comment 1 Ryan Phillips 2020-02-25 15:48:49 UTC
*** Bug 1771016 has been marked as a duplicate of this bug. ***

Comment 2 Ryan Phillips 2020-02-25 15:49:49 UTC
*** Bug 1776185 has been marked as a duplicate of this bug. ***

Comment 3 Ryan Phillips 2020-03-12 13:32:53 UTC
*** Bug 1797828 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.