Description of problem: The monitoring system does not make any distinction between pods that are in the host network and pods that aren't. As a pod in the host network does not have its own networking stack but just views the host's monitoring stack, the node metrics are accounted. This can lead to some issues, like: - Node metrics are shown as if they were the individual metrics of some pods, which can be confusing - If the networking metrics of all the projects are summed to get a total consumption (like management console does), for each host network pod, the metrics of the node where it resides are summed as if the metrics of an individual pod were, completely distorting the metric (for each host network pod, the metrics of the node where it resides are summed). So best solution would be to not account host network pods in the metrics. Version-Release number of selected component (if applicable): 4.4 How reproducible: Always Steps to Reproduce: 1. Look at networking metrics of any pod in the host network, like many system pods. 2. 3. Actual results: Host network pod metrics are included and summed as if they were individual pods with their own networking stack, causing distortion. Expected results: Only account metrics for pods that are not in the host network. Additional info:
There are a couple of things mixed here I think let's take them apart. I think the initial observation of the cluster dashboard showing incorrect data is correct, this should be fixed. Since as you mentioned the container level metrics expose whatever interface they have available, which works as intended. I do agree that this can be confusing, I will ask upstream cAdvisor if this may be reconsidered, but I am not hopeful as all of cAdvisor works this way (I opened this upstream issue: https://github.com/google/cadvisor/issues/2615). My suggestion for a fix of the console would be to have the console not sum up the container metrics, but rather the node level metrics, which would be these recording rules * instance:node_network_receive_bytes_excluding_lo:rate1m * instance:node_network_transmit_bytes_excluding_lo:rate1m As this would be done by console, I'm going to transfer this bugzilla to the console component.
Thanks. It is important to ensure that, if we sume node-level metrics, all the traffic from pods is still accounted. I mean: If 2 pods on the same node communicate with each other, that traffic may not be seen in hosts main interfaces (as there would be no need to send anything through vxlan).
I think you have a point, but I don't see how we could combine that into a single graph, any suggestions? I would propose that the graph we have is about total cluster level network traffic which node network statistics would be most representative for. I think it would be valid to open an RFE for an *additional* graph being all pod network traffic (including non vxlan traffic). Thoughts?
As long as we properly clarify that it is only node-level traffic, I can agree. The rest of what I suggested may require a RFE, as it may require some degree of redesign.
Shouldnt we also adjust Top Consumer queries ? Right now we use topk(25, sort_desc(sum(rate(container_network_receive_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace))) topk(25, sort_desc(sum(rate(container_network_transmit_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace)))
Queries are: Top Project consumers IN: topk(25, sort_desc(sum(rate(container_network_receive_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace))) OUT: topk(25, sort_desc(sum(rate(container_network_transmit_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace))) Top Pod consumers IN: topk(25, sort_desc(sum(rate(container_network_receive_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace, pod))) OUT: topk(25, sort_desc(sum(rate(container_network_transmit_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace, pod))) Top Node consumers IN: topk(25, sort_desc(sum(rate(container_network_receive_bytes_total{ container="POD", pod!= ""}[5m])) BY (node))) OUT: topk(25, sort_desc(sum(rate(container_network_transmit_bytes_total{ container="POD", pod!= ""}[5m])) BY (node)))
Yes those queries are exhibiting the same problem. Top project and top pod are not possible with the current set of metrics. Top node would be fixed with the same metrics/recording-rules that I referenced earlier.
verification blocked by bz 1862885
Now cluster utilization query is: By Node: Network in breakdown query is: topk(25, sort_desc(sum(instance:node_network_receive_bytes_excluding_lo:rate1m) BY (instance))) Network out breakdown query is: topk(25, sort_desc(sum(instance:node_network_transmit_bytes_excluding_lo:rate1m) BY (instance))) By Project: Network in breakdown query is: topk(25, sort_desc(sum(rate(container_network_receive_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace))) Network out breakdown query is: topk(25, sort_desc(sum(rate(container_network_transmit_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace))) By Pods: Network in breakdown query is: topk(25, sort_desc(sum(rate(container_network_receive_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace, pod))) Network out breakdown query is: topk(25, sort_desc(sum(rate(container_network_transmit_bytes_total{ container="POD", pod!= ""}[5m])) BY (namespace, pod))) By Node metrics query is using node_network_receive_bytes_excluding_lo & node_network_transmit_bytes_excluding_lo as suggested. Verified on 4.6.0-0.nightly-2020-08-23-185640 Let me know if this is wrong.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196