Description of problem: Based on prometheus queries, it appears that metrics being used by online components (e.g. container_network_receive_bytes_total) to assess user activity are not being collected for standard user projects. The metrics are recorded for openshift-*, kube-system, and the like, but are not present for arbitrary user projects. Version-Release number of selected component (if applicable): 3.11.0-0.21.0 Actual results: No time series of container_network_receive_bytes_total for non core projects. Expected results: container_network_receive_bytes_total would exist in prometheus for all projects.
Can you clarify which Prometheus environment you're querying? When I query this on the central cluster: ``` sum(container_network_receive_bytes_total) by(namespace) ``` I see all namespaces, not just the cluster component namespaces.
You're right.. there do appear to be some non-core namespaces. I was expecting to see far more. oc get pods --all-namespaces | grep Running | cut -f 1 -d ' ' | uniq | wc Results in 6329 atm on ca-central-1. However, in prometheus, your query results in 161 time series.
Looking at the cAdvisor code, it may just be that containers from other namespaces simply have not received any network traffic: https://github.com/google/cadvisor/blob/be9cd931d724ef2aaa88e7f255ea38ebd904eaa0/metrics/prometheus.go#L633-L640