Created attachment 1212537 [details] container memory usage graph Description of problem: Containers are given 2Gi memory limit but metrics show over usage > 2G See attached template file Version-Release number of selected component (if applicable): - Hack day env (Oct 19, 2016) How reproducible: Steps to Reproduce: 1. Upload attached template to your porject 2. oc new-app --template=hawkular-full Actual results: Metric graph shows mem usage > resource limit Expected results: OpenShift should restart containers once memory consumption exceeds resource limit Additional info:
Created attachment 1212538 [details] free cmd inside container
Created attachment 1212539 [details] test template
Note that free doesn't report the value inside the container. It's reporting node level metrics.
Seth - can you try to reproduce using the specified template to see if the actually produced pod looks as expected?
I was able to reproduce. # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 901ee01d8893 docker.io/hawkularqe/cassandra:latest "/docker-start.sh" 5 minutes ago Up 5 minutes k8s_hawkular-full-cnode.61f275f8_hawkular-full-1-8k5rs_demo_9338c2d7-a079-11e6-a193-fa163e4dab0a_4466a9bb 45567860dd7b docker.io/hawkularqe/hawkular-services:latest "/bin/bash -c ${JBOSS" 5 minutes ago Up 5 minutes k8s_hawkular-full.bd21ab94_hawkular-full-1-8k5rs_demo_9338c2d7-a079-11e6-a193-fa163e4dab0a_484ffbd0 There are two containers in the pod: -bash-4.2# cd /sys/fs/cgroup/memory/system.slice/docker-901ee01d8893d24f15f674f9a086514a7ba5ab887c1d5d2a628d20c3549ee17c.scope/ -bash-4.2# cat memory.limit_in_bytes 2147483648 -bash-4.2# cat memory.usage_in_bytes 1473753088 -bash-4.2# cd /sys/fs/cgroup/memory/system.slice/docker-45567860dd7b191f1f15b19f284d3403c46a1658348d9d370c5639a6e843354a.scope/ -bash-4.2# cat memory.limit_in_bytes 2147483648 -bash-4.2# cat memory.usage_in_bytes 1669681152 This is an issue with the web console. requests are made at the container-level, however the web console is using the container-level limit with the pod-level usage. It should use the container-level usage of the container selected in the dropdown.
Moving this to the console team.
(In reply to Seth Jennings from comment #5) > requests are made at the container-level, however the web console is using > the container-level limit with the pod-level usage. It should use the > container-level usage of the container selected in the dropdown. Seth, Derek, we request container-level metrics for memory usage from Hawkular. We should not be displaying pod-level metrics, which is why we have the container dropdown. Do you see different values if you switch the selected container in the dropdown?
I believe I see the problem. This does look like a web console bug.
https://github.com/openshift/origin-web-console/pull/792
Moving this to OCP since it is not a bug specific to online
Reproduced the issue on OCP 3.3. Tested on OCP 3.4 with version: v3.4.0.28+dfe3a66. Created an app of which the pod has 2 containers, set memory limits to each container, check metrics of each container, when container memory limits is reached, there will be no available memory for the container. Attached the screeshots. The bug has been fixed, so move it to verified.
Created attachment 1222344 [details] container1-memory.png
Created attachment 1222346 [details] container2-memory.png
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066