Description of problem: The metering reporting-operator is not importing metrics for container cpu/memory usage metrics, but is for other types of metrics. Version-Release number of selected component (if applicable): 4.3.x How reproducible: Always Steps to Reproduce: 1. Querying Prometheus directly shows the same behavior. sum(rate(container_cpu_usage_seconds_total{container_name!="POD",container_name!="",pod!=""}[1m])) BY (pod, namespace) + on (pod, namespace) group_left(node) (sum(kube_pod_info{pod_ip!="",node!="",host_ip!=""}) by (pod, namespace, node) * 0) and sum(container_memory_usage_bytes{container_name!="POD", container_name!="",pod!=""}) by (pod, namespace) + on (pod, namespace) group_left(node) (sum(kube_pod_info{pod_ip!="",node!="",host_ip!=""}) by (pod, namespace, node) * 0) both return no metrics in 4.3, but work in 4.2 and 4.1 After investigation it's because the container_name metric label changed to container in Kube 1.14, and in 1.16 the old metric labels such as container_name and pod_name were removed. We need to update our metrics queries to use container instead of container_name.
verified with 4.3.0-0.nightly-2019-10-22-101148 and `metering` master branch 1. install metering 4.2 using manual step 2. get reportdatasource and see missing import meterics pruan@MacBook-Pro ~/workspace/gocode/src/github.com/operator-framework/operator-metering (release-4.2●)$ oc get reportdatasource [ruby-2.6.3] NAME EARLIEST METRIC NEWEST METRIC IMPORT START IMPORT END LAST IMPORT TIME AGE cluster-cpu-capacity-raw 9m33s cluster-cpu-usage-raw 9m33s cluster-memory-capacity-raw 9m33s cluster-memory-usage-raw 9m33s node-allocatable-cpu-cores 2019-10-22T18:47:00Z 2019-10-22T19:58:00Z 2019-10-22T18:47:00Z 2019-10-22T19:58:00Z 2019-10-22T20:49:38Z 9m33s node-allocatable-memory-bytes 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T20:49:42Z 9m33s node-capacity-cpu-cores 2019-10-22T18:47:00Z 2019-10-22T19:58:00Z 2019-10-22T18:47:00Z 2019-10-22T19:58:00Z 2019-10-22T20:49:36Z 9m33s node-capacity-memory-bytes 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T20:49:42Z 9m33s node-cpu-allocatable-raw 9m33s node-cpu-capacity-raw 9m33s node-memory-allocatable-raw 9m33s node-memory-capacity-raw 9m33s persistentvolumeclaim-capacity-bytes 2019-10-22T18:47:00Z 2019-10-22T19:55:00Z 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T20:49:38Z 9m33s persistentvolumeclaim-capacity-raw 9m33s persistentvolumeclaim-phase 2019-10-22T18:47:00Z 2019-10-22T19:56:00Z 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T20:49:41Z 9m33s persistentvolumeclaim-phase-raw 9m33s persistentvolumeclaim-request-bytes 2019-10-22T18:47:00Z 2019-10-22T19:52:00Z 2019-10-22T18:47:00Z 2019-10-22T19:52:00Z 2019-10-22T20:49:35Z 9m33s persistentvolumeclaim-request-raw 9m33s persistentvolumeclaim-usage-bytes 2019-10-22T18:47:00Z 2019-10-22T20:16:00Z 2019-10-22T18:47:00Z 2019-10-22T20:16:00Z 2019-10-22T20:49:45Z 9m33s persistentvolumeclaim-usage-raw 9m33s persistentvolumeclaim-usage-with-phase-raw 9m33s pod-cpu-request-raw 9m33s pod-cpu-usage-raw 9m33s pod-limit-cpu-cores 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T20:49:39Z 9m33s pod-limit-memory-bytes 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T18:47:00Z 2019-10-22T20:04:00Z 2019-10-22T20:49:44Z 9m33s pod-memory-request-raw 9m33s pod-memory-usage-raw 9m33s pod-persistentvolumeclaim-request-info 2019-10-22T18:47:00Z 2019-10-22T19:56:00Z 2019-10-22T18:47:00Z 2019-10-22T19:58:00Z 2019-10-22T20:49:40Z 9m33s pod-request-cpu-cores 2019-10-22T18:47:00Z 2019-10-22T19:52:00Z 2019-10-22T18:47:00Z 2019-10-22T19:52:00Z 2019-10-22T20:49:38Z 9m33s pod-request-memory-bytes 2019-10-22T18:47:00Z 2019-10-22T19:52:00Z 2019-10-22T18:47:00Z 2019-10-22T19:52:00Z 2019-10-22T20:49:34Z 9m33s pod-usage-cpu-cores 2019-10-22T18:47:00Z 2019-10-22T20:22:00Z 2019-10-22T20:49:44Z 9m33s pod-usage-memory-bytes 2019-10-22T18:47:00Z 2019-10-22T20:16:00Z 2019-10-22T20:49:37Z 9m33s 3. git checkout master 4. run ./hack/opnenshift-install.sh to upgrade to 4.3 5. wait until the upgrade is completed and do `oc get repordatasource` again pod-usage-cpu-cores 2019-10-22T19:32:00Z 2019-10-22T21:37:00Z 2019-10-22T19:19:00Z 2019-10-22T21:37:00Z 2019-10-22T21:41:44Z 29m pod-usage-memory-bytes 2019-10-22T19:32:00Z 2019-10-22T21:37:00Z 2019-10-22T19:19:00Z 2019-10-22T21:37:00Z 2019-10-22T21:41:48Z 29m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062