Attachment 1541234 Details for Bug 1685807 – "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod

"many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod

matching_not_allowed.txt (text/plain), 40.12 KB, created by Junqi Zhao on 2019-03-06 07:18:46 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Junqi Zhao

Created: 2019-03-06 07:18:46 UTC

Size: 40.12 KB

patch

obsolete

>#  oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "Evaluating rule failed"
>level=warn ts=2019-03-06T06:20:25.743554993Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:25.779452969Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:25.782227903Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:25.785908382Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:55.76933694Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:55.77732702Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:55.780930804Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:20:55.785162357Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:25.751474294Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:25.756604745Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:25.759139157Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:25.763889957Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:55.745202913Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:55.749581537Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:55.752489597Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:21:55.756285944Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:23:55.748847422Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:23:55.754063282Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:23:55.758188326Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:23:55.765724307Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.306038368Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.308264628Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.309589743Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.311533654Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.312526737Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n  group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.314704376Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.316179066Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.317664511Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.319082778Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.326432133Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:05.330993516Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n  on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:25.747760476Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:25.753177351Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:25.756760513Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:25.761440766Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.301591643Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.303342463Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.304533025Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.306413856Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.3073017Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n  group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.309468511Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.310850512Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.312328269Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.313708752Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.32111086Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:35.325739749Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n  on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:55.745613457Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:55.750822461Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:55.754207311Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:24:55.759563478Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.302921931Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.304785608Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.306133748Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.308212355Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.309195324Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n  group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.311299752Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.312704522Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.314314186Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.315662455Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.322614981Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:05.327633294Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n  on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:25.745568908Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:25.750302023Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:25.753108417Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:25.756948368Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n  and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n  \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.302323173Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.304084056Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n  pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.305399802Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.307384367Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.308381003Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n  group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.310728703Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n  + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.311991027Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.313407529Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.314844638Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.321459766Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n  * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>level=warn ts=2019-03-06T06:25:35.325878434Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n  + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n  on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
>

Actions: View

Attachments on bug 1685807: 1541234 | 1601348