Description of problem: We would like to document/test the use of the 'additionalScrapeConfigs:' to provide additional datasets to custom grafana dashboards. The specific use-case here would be to have a grafana dashboard that displays CPU/Memory statistics on a set of nodes with a specific label.
i.e
Grafana dashboard that shows nodes X, Y, and Z CPU/Memory statistics with key/value pair label=blah
Currently, we can make a promsql query and get node info and labels, but the CPU/Memory metrics do not have an associated node label value.
Version-Release number of selected component (if applicable):
3.11 latest
Actual results: Currently this procedure/feature is not supported or documented:
https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html#configuring-openshift-cluster-monitoring
Expected results: We would like for additional grafana dashboards to cover this use-case.
Additional info:
Comment 1Frederic Branczyk
2019-02-04 14:27:50 UTC
What you are requesting is really two concerns. 1) Modifying the Prometheus config and 2) Adding dashboards. For your case you really doesn't need modification of the Prometheus config, as the metadata you are intending to add, should not be on the target, but should be added at query time using a join. So really all you need is the ability to add custom dashboards, which as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1647932, is something we are looking into but have no news on when/if this will ship.
Comment 2Christian Heidenreich
2019-02-05 13:43:03 UTC
*** Bug 1622591 has been marked as a duplicate of this bug. ***
I had to do some digging to figure out how to do this, so I'm sharing my knowledge here.
How to do joins in PromQL https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching
Change queries like
1 - avg(rate(node_cpu{mode="idle"}[1m]))
to
1 - avg(rate(node_cpu{mode="idle"}[1m]) * ON (instance) group_left(node) label_replace(node_uname_info, "node", "$1", "nodename", "(.*)") * ON(node) group_left() kube_node_labels{label_region="app"})
to restrict to region=app nodes.
Created attachment 1542176[details]
Modified "K8s / Compute Resources / Cluster" dashboard that limits all metrics to nodes that have the region=app label.
Description of problem: We would like to document/test the use of the 'additionalScrapeConfigs:' to provide additional datasets to custom grafana dashboards. The specific use-case here would be to have a grafana dashboard that displays CPU/Memory statistics on a set of nodes with a specific label. i.e Grafana dashboard that shows nodes X, Y, and Z CPU/Memory statistics with key/value pair label=blah Currently, we can make a promsql query and get node info and labels, but the CPU/Memory metrics do not have an associated node label value. Version-Release number of selected component (if applicable): 3.11 latest Actual results: Currently this procedure/feature is not supported or documented: https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html#configuring-openshift-cluster-monitoring Expected results: We would like for additional grafana dashboards to cover this use-case. Additional info: