Bug 1911173 - [MSTR-998] Many charts' legend names show {{}} instead of words
Summary: [MSTR-998] Many charts' legend names show {{}} instead of words
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.7
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.10.0
Assignee: Andrew Pickering
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-28 04:07 UTC by Xingxing Xia
Modified: 2022-03-10 16:02 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:02:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Ctrl + F finds 19 {{ matches of wrong legend (51.41 KB, image/png)
2020-12-28 04:13 UTC, Xingxing Xia
no flags Details
grafana-rate.png (134.76 KB, image/png)
2021-02-05 17:03 UTC, Abu Kashem
no flags Details
console-rate.png (86.22 KB, image/png)
2021-02-05 17:04 UTC, Abu Kashem
no flags Details
console-watchers.png (96.10 KB, image/png)
2021-02-05 17:10 UTC, Abu Kashem
no flags Details
grafana-watchers.png (133.28 KB, image/png)
2021-02-05 17:11 UTC, Abu Kashem
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift console pull 10655 0 None open Bug 1911173: Monitoring dashboards: Improve series titles when a label is missing 2021-12-14 00:52:53 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:02:50 UTC

Description Xingxing Xia 2020-12-28 04:07:12 UTC
Description of problem:
Many charts' legend names show {{}} instead of words

Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2020-12-21-131655

How reproducible:
Always

Steps to Reproduce:
1. Login to management console, navigate Monitoring --> Dashboards.
In the Dashboard drop-down list, select “API Performance”.
2. Use Ctrl + F to search "{{"

Actual results:
2. The browser shows 19 matches. E.g.
{{component}}-{{resource}} under "requests terminated rate" chart.
{{flowSchema}}:{{priorityLevel}} under chart "p&f - request wait duration - 99th quantile"
... and so on many many ...
See screenshort.

Expected results:
2. Ctrl + F should result in such things. 


Additional info:
Take "p&f - request wait duration - 99th quantile" for example, the configmap shows its "expr" uses apiserver_flowcontrol_request_wait_duration_seconds_bucket, I query this in Prometheus, got:
apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver="kube-apiserver", endpoint="https", execute="true", flow_schema="catch-all", instance="10.0.141.180:6443", job="apiserver", le="+Inf", namespace="default", priority_level="catch-all", service="kubernetes"}.

So the cause seems to be:
The definition spells wrong for some fields, e.g. they're flow_schema and priority_level, instead of flowSchema and priorityLevel.

Comment 1 Xingxing Xia 2020-12-28 04:13:21 UTC
Created attachment 1742500 [details]
Ctrl + F finds 19 {{ matches of wrong legend

Comment 2 Abu Kashem 2021-01-04 20:05:00 UTC
Hi xxia,
upstream changed the label names for the p&f metrics, that's why you see these unresolved label values. The following PR https://github.com/openshift/cluster-kube-apiserver-operator/pull/1024 should fix the issue. 

Also, a linked BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1910140 I had created before you reported this issue.

Comment 3 Xingxing Xia 2021-01-05 01:52:58 UTC
OK, didn't notice that, feel free to close one of them as DUP, thanks.

Comment 4 Xingxing Xia 2021-01-18 12:03:27 UTC
Per bug 1910140#c2 and its following attachment, this bug has more {{, so it is different issue. Will not mark this as DUP.

Comment 5 Abu Kashem 2021-02-03 22:49:53 UTC
xxia

this seems to be an issue with the console dashboard. the same query in grafana works fine.

Comment 6 hongyan li 2021-02-04 03:44:50 UTC
This is an console dashboard issue, fix it directly, as grafana will be deprecated soon, don't need check.

Comment 7 Xingxing Xia 2021-02-04 06:03:09 UTC
> the same query in grafana works fine.
Tried to double confirm this. But no idea how to, due to bug 1911182. Also tried to consult Monitoring QE, they say no idea how to, either.
Hmm, therefore moving component to Management Console for triage without further doulbe-check exploring.

Comment 8 Andrew Pickering 2021-02-04 13:58:34 UTC
The text being displayed is the `legendFormat` template string from the dashboard definition. This template string is displayed when the template cannot be interpolated. (The `{{...}}` strings represent variables that should be interpolated.)

It looks like this is happening because some of the variables defined in the template strings do not exist in the metric labels.

The same text is also visible in the graph tooltips.

Since this is limited to the legend and tooltip text, I'm setting the severity to low.

Comment 9 Andrew Pickering 2021-02-04 14:24:35 UTC
Confirmed that this seems to only be an issue on the "API Performance" dashboard.

Comment 10 Andrew Pickering 2021-02-05 02:16:24 UTC
I'm seeing just 2 variables that don't exist (`{{resource}}` and `{{group}}`), causing the error. Both errors are seen a number of times and for more than one graph.

For example, this query returns one result with no `resource` label value:
`topk(20, sum(rate(apiserver_request_total{apiserver="kube-apiserver"}[5m])) by(resource,verb))`
(`legendFormat` is `{{resource}}-{{verb}}`)

And this query returns a few results with no `group` label value:
`topk(25, sum(apiserver_registered_watchers{apiserver="kube-apiserver"}) by(group,kind))`
(`legendFormat` is `{{group}}:{{kind}}`)

Comment 11 Andrew Pickering 2021-02-05 11:47:35 UTC
Assigning back kube-apiserver component since it seems the problem is with the dashboard definition `legendFormat` strings.

Could you please look at this again Abu?

Comment 12 Abu Kashem 2021-02-05 17:03:10 UTC
Created attachment 1755269 [details]
grafana-rate.png

Comment 13 Abu Kashem 2021-02-05 17:04:46 UTC
Created attachment 1755270 [details]
console-rate.png

Comment 14 Abu Kashem 2021-02-05 17:10:35 UTC
Created attachment 1755271 [details]
console-watchers.png

Comment 15 Abu Kashem 2021-02-05 17:11:04 UTC
Created attachment 1755272 [details]
grafana-watchers.png

Comment 16 Abu Kashem 2021-02-05 17:20:39 UTC
> I'm seeing just 2 variables that don't exist (`{{resource}}` and `{{group}}`), causing the error. Both errors are seen a number of times and for more than one graph.

>For example, this query returns one result with no `resource` label value:
> `topk(20, sum(rate(apiserver_request_total{apiserver="kube-apiserver"}[5m])) by(resource,verb))`
> (`legendFormat` is `{{resource}}-{{verb}}`)

So I loaded the query on a grafana instance - grafana shows it as "-GET" and console shows it as "{{resource}}-{{verb}}"
Look at https://bugzilla.redhat.com/attachment.cgi?id=1755269 https://bugzilla.redhat.com/attachment.cgi?id=1755270


> And this query returns a few results with no `group` label value:
> `topk(25, sum(apiserver_registered_watchers{apiserver="kube-apiserver"}) by(group,kind))`
> (`legendFormat` is `{{group}}:{{kind}}`)

Look at https://bugzilla.redhat.com/attachment.cgi?id=1755271 and https://bugzilla.redhat.com/attachment.cgi?id=1755272


looks like console just shows the field legend template when one of the labels is missing while grafana does a best-effort.
I am moving this to the console team for now, please look at it and let us know if you have any feedback.

Comment 17 Andrew Pickering 2021-02-06 07:17:08 UTC
Thanks Abu

I agree that Console should probably be updated to do a better job of displaying the legend strings in this case, but isn't the issue also that the `legendFormat` strings are referencing variables that don't exist?

Comment 18 Jakub Hadvig 2021-02-09 14:27:53 UTC
Abu, could you please check Andy's question in #c17

Comment 19 Abu Kashem 2021-02-09 21:11:05 UTC
> I agree that Console should probably be updated to do a better job of displaying the legend strings in this case, but isn't the issue also that the `legendFormat` strings are referencing variables that don't exist?

to my knowledge, 'legendFormat' strings are not referencing any variable, it is referencing labels. you can check https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/manifests/0000_90_kube-apiserver-operator_05_api_performance_dashboard.yaml, and please let me know where you see 'legendFormat' referencing variables.

Comment 20 Martin Bukatovic 2021-04-19 17:54:56 UTC
I just noticed this in Dashboards page of OCP Console with OCP 4.7.0-0.nightly-2021-04-15-110345

- "{{component}}-{{resource}}" in Requests Terminated Rate char
- "{{resource}}-{{verb}}" in Request Rate by Resource and Verb charts
- "{{group}}:{{kind}}" in Registered Watchers chart

Comment 26 Junqi Zhao 2021-12-14 13:32:11 UTC
tested with the PR, "API Performance" dashboard, no {{component}}-{{resource}} show in the graph, see from the picture

Comment 37 errata-xmlrpc 2022-03-10 16:02:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.