Login
Log in using an SSO provider:
Fedora Account System
Red Hat Associate
Red Hat Customer
Login using a Red Hat Bugzilla account
Forgot Password
Create an Account
Red Hat Bugzilla – Attachment 1541234 Details for
Bug 1685807
[3.11]Sometimes meet "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod
Home
New
Search
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh92 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
[?]
This site requires JavaScript to be enabled to function correctly, please enable it.
"many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod
matching_not_allowed.txt (text/plain), 40.12 KB, created by
Junqi Zhao
on 2019-03-06 07:18:46 UTC
(
hide
)
Description:
"many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod
Filename:
MIME Type:
Creator:
Junqi Zhao
Created:
2019-03-06 07:18:46 UTC
Size:
40.12 KB
patch
obsolete
># oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "Evaluating rule failed" >level=warn ts=2019-03-06T06:20:25.743554993Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:25.779452969Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:25.782227903Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:25.785908382Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:55.76933694Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:55.77732702Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:55.780930804Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:20:55.785162357Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:25.751474294Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:25.756604745Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:25.759139157Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:25.763889957Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:55.745202913Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:55.749581537Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:55.752489597Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:21:55.756285944Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:23:55.748847422Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:23:55.754063282Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:23:55.758188326Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:23:55.765724307Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.306038368Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.308264628Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.309589743Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.311533654Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.312526737Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.314704376Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.316179066Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.317664511Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.319082778Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.326432133Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:05.330993516Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:25.747760476Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:25.753177351Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:25.756760513Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:25.761440766Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.301591643Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.303342463Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.304533025Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.306413856Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.3073017Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.309468511Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.310850512Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.312328269Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.313708752Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.32111086Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:35.325739749Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:55.745613457Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:55.750822461Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:55.754207311Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:24:55.759563478Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.302921931Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.304785608Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.306133748Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.308212355Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.309195324Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.311299752Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.312704522Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.314314186Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.315662455Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.322614981Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:05.327633294Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:25.745568908Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:25.750302023Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:25.753108417Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:25.756948368Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n and on(pod) kube_pod_status_scheduled{condition=\"true\"}) * on(namespace, pod) group_left(label_name)\n label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\",\n \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.302323173Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(node) (sum by(node, cpu) (node_cpu{job=\"node-exporter\"} * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.304084056Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_cpu_utilisation:avg1m\nexpr: 1 - avg by(node) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[1m]) * on(namespace,\n pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.305399802Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_cpu_saturation_load1:'\nexpr: sum by(node) (node_load1{job=\"node-exporter\"} * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / node:node_num_cpu:sum\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.307384367Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_available:sum\nexpr: sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.308381003Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_bytes_total:sum\nexpr: sum by(node) (node_memory_MemTotal{job=\"node-exporter\"} * on(namespace, pod)\n group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.310728703Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: 'node:node_memory_utilisation:'\nexpr: 1 - sum by(node) ((node_memory_MemFree{job=\"node-exporter\"} + node_memory_Cached{job=\"node-exporter\"}\n + node_memory_Buffers{job=\"node-exporter\"}) * on(namespace, pod) group_left(node)\n node_namespace_pod:kube_pod_info:) / sum by(node) (node_memory_MemTotal{job=\"node-exporter\"}\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.311991027Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_memory_swap_io_bytes:sum_rate\nexpr: 1000 * sum by(node) ((rate(node_vmstat_pgpgin{job=\"node-exporter\"}[1m]) + rate(node_vmstat_pgpgout{job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.313407529Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_utilisation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_ms{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.314844638Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_disk_saturation:avg_irate\nexpr: avg by(node) (irate(node_disk_io_time_weighted{device=~\"(sd|xvd|nvme).+\",job=\"node-exporter\"}[1m])\n / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.321459766Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_utilisation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_bytes{device!~\"veth.+\",job=\"node-exporter\"}[1m]))\n * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >level=warn ts=2019-03-06T06:25:35.325878434Z caller=manager.go:343 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_net_saturation:sum_irate\nexpr: sum by(node) ((irate(node_network_receive_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])\n + irate(node_network_transmit_drop{device!~\"veth.+\",job=\"node-exporter\"}[1m])) *\n on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)\n" err="many-to-many matching not allowed: matching labels must be unique on one side" >
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1685807
: 1541234 |
1601348