Bug 1319700

3. What is the nature and description of the request? 
When the OutOfMemoryKiller of RHEL k01602613ills processes on a node (e.g. a pod is killed and not running anymore), this should be reflected in 'oc get pod --all-namspaces -o wide', now it shows running, although it is down.


As an admin of the cluster, if my nodes are overloaded pods are gradually spread to other nodes.

Doing some test using nexus:
- oc new-project test-x
- oc new-app sonatype/nexus
- (strategy BestEffort, no resource requests, no limits)

As the footprint of nexus is higher, I reached the limit with regards to memory, for the node I was using, 32 GB of memory are available.

In the end (at around 55-60 pods), I was in the following situation:
- most of the pods were not running anymore, as the OutOfMemoryKiller of Linux was killing processes
(/var/log/messages showed many of Mar 16 13:45:32 ip-10-191-1-114 kernel: Out of memory: Kill process 120303 (java) score 1016 or sacrifice child)
- docker was in an unusable state
(docker ps was not showing anything anymore, not returning a result, deleting a pod ended in the pod staying on Terminating, docker restart required...)
- OpenShift (or kubernetes) still showed that everything was running
(oc get pod --all-namespaces -o wide -> all pods Running)

So the environment got into an unstable state. This should not be possible.

Issue1
oc get pod --all-namespaces -o wide should reflect the truth. Although pods were not running anymore, the command shows that everything is running.

Issue2
It should not be possible to schedule a pod to a node, when no more resources are available (e.g. no more memory or 90% of the available memory are already used or  Verify that a cluster at high utilization (> 90%) sees pods moved to a newly added node after rebalancing occurs


4. Why does the customer need this? (List the business requirements here)

Nobody would recognize/expect that a pod is down, if it still shows running. Effective management of the cluster with proper evacuation of the pods based on the resources of the cluster

5. How would the customer like to achieve this? (List the functional requirements here) 
I don't know the details of current implementation, but I guess some polling on the pods should be done, to see if etcd corresponds with real life.

6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. 
On a node with 32GB memory, create about 55-60 projects with a running nexus (oc new-app sonatype/nexus). When the OOM kicks in, check what oc get pod shows.

7. Is there already an existing RFE upstream or in Red Hat Bugzilla? 

No

8. Does the customer have any specific timeline dependencies and which release would they like to target ? 
asap

9. Is the sales team involved in this request and do they have any additional input? Red Hat Consultant on site, account team fully aware of the request. 
?

10. List any affected packages or components. 
?

11. Would the customer be able to assist in testing this functionality if implemented? 
yes