Description of problem: Many charts' legend names show {{}} instead of words Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2020-12-21-131655 How reproducible: Always Steps to Reproduce: 1. Login to management console, navigate Monitoring --> Dashboards. In the Dashboard drop-down list, select “API Performance”. 2. Use Ctrl + F to search "{{" Actual results: 2. The browser shows 19 matches. E.g. {{component}}-{{resource}} under "requests terminated rate" chart. {{flowSchema}}:{{priorityLevel}} under chart "p&f - request wait duration - 99th quantile" ... and so on many many ... See screenshort. Expected results: 2. Ctrl + F should result in such things. Additional info: Take "p&f - request wait duration - 99th quantile" for example, the configmap shows its "expr" uses apiserver_flowcontrol_request_wait_duration_seconds_bucket, I query this in Prometheus, got: apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver="kube-apiserver", endpoint="https", execute="true", flow_schema="catch-all", instance="10.0.141.180:6443", job="apiserver", le="+Inf", namespace="default", priority_level="catch-all", service="kubernetes"}. So the cause seems to be: The definition spells wrong for some fields, e.g. they're flow_schema and priority_level, instead of flowSchema and priorityLevel.
Created attachment 1742500 [details] Ctrl + F finds 19 {{ matches of wrong legend
Hi xxia, upstream changed the label names for the p&f metrics, that's why you see these unresolved label values. The following PR https://github.com/openshift/cluster-kube-apiserver-operator/pull/1024 should fix the issue. Also, a linked BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1910140 I had created before you reported this issue.
OK, didn't notice that, feel free to close one of them as DUP, thanks.
Per bug 1910140#c2 and its following attachment, this bug has more {{, so it is different issue. Will not mark this as DUP.
xxia this seems to be an issue with the console dashboard. the same query in grafana works fine.
This is an console dashboard issue, fix it directly, as grafana will be deprecated soon, don't need check.
> the same query in grafana works fine. Tried to double confirm this. But no idea how to, due to bug 1911182. Also tried to consult Monitoring QE, they say no idea how to, either. Hmm, therefore moving component to Management Console for triage without further doulbe-check exploring.
The text being displayed is the `legendFormat` template string from the dashboard definition. This template string is displayed when the template cannot be interpolated. (The `{{...}}` strings represent variables that should be interpolated.) It looks like this is happening because some of the variables defined in the template strings do not exist in the metric labels. The same text is also visible in the graph tooltips. Since this is limited to the legend and tooltip text, I'm setting the severity to low.
Confirmed that this seems to only be an issue on the "API Performance" dashboard.
I'm seeing just 2 variables that don't exist (`{{resource}}` and `{{group}}`), causing the error. Both errors are seen a number of times and for more than one graph. For example, this query returns one result with no `resource` label value: `topk(20, sum(rate(apiserver_request_total{apiserver="kube-apiserver"}[5m])) by(resource,verb))` (`legendFormat` is `{{resource}}-{{verb}}`) And this query returns a few results with no `group` label value: `topk(25, sum(apiserver_registered_watchers{apiserver="kube-apiserver"}) by(group,kind))` (`legendFormat` is `{{group}}:{{kind}}`)
Assigning back kube-apiserver component since it seems the problem is with the dashboard definition `legendFormat` strings. Could you please look at this again Abu?
Created attachment 1755269 [details] grafana-rate.png
Created attachment 1755270 [details] console-rate.png
Created attachment 1755271 [details] console-watchers.png
Created attachment 1755272 [details] grafana-watchers.png
> I'm seeing just 2 variables that don't exist (`{{resource}}` and `{{group}}`), causing the error. Both errors are seen a number of times and for more than one graph. >For example, this query returns one result with no `resource` label value: > `topk(20, sum(rate(apiserver_request_total{apiserver="kube-apiserver"}[5m])) by(resource,verb))` > (`legendFormat` is `{{resource}}-{{verb}}`) So I loaded the query on a grafana instance - grafana shows it as "-GET" and console shows it as "{{resource}}-{{verb}}" Look at https://bugzilla.redhat.com/attachment.cgi?id=1755269 https://bugzilla.redhat.com/attachment.cgi?id=1755270 > And this query returns a few results with no `group` label value: > `topk(25, sum(apiserver_registered_watchers{apiserver="kube-apiserver"}) by(group,kind))` > (`legendFormat` is `{{group}}:{{kind}}`) Look at https://bugzilla.redhat.com/attachment.cgi?id=1755271 and https://bugzilla.redhat.com/attachment.cgi?id=1755272 looks like console just shows the field legend template when one of the labels is missing while grafana does a best-effort. I am moving this to the console team for now, please look at it and let us know if you have any feedback.
Thanks Abu I agree that Console should probably be updated to do a better job of displaying the legend strings in this case, but isn't the issue also that the `legendFormat` strings are referencing variables that don't exist?
Abu, could you please check Andy's question in #c17
> I agree that Console should probably be updated to do a better job of displaying the legend strings in this case, but isn't the issue also that the `legendFormat` strings are referencing variables that don't exist? to my knowledge, 'legendFormat' strings are not referencing any variable, it is referencing labels. you can check https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/manifests/0000_90_kube-apiserver-operator_05_api_performance_dashboard.yaml, and please let me know where you see 'legendFormat' referencing variables.
I just noticed this in Dashboards page of OCP Console with OCP 4.7.0-0.nightly-2021-04-15-110345 - "{{component}}-{{resource}}" in Requests Terminated Rate char - "{{resource}}-{{verb}}" in Request Rate by Resource and Verb charts - "{{group}}:{{kind}}" in Registered Watchers chart
tested with the PR, "API Performance" dashboard, no {{component}}-{{resource}} show in the graph, see from the picture
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056