Bug 2020522

Summary: 400 Bad Request error for the metrics result
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: Dev ConsoleAssignee: Sahil Budhwar <sbudhwar>
Status: CLOSED DUPLICATE QA Contact: spathak <spathak>
Severity: high Docs Contact:
Priority: high    
Version: 4.10CC: aos-bugs, msaud, nmukherj, sbudhwar
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-01 10:00:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
400 Bad Request error none

Description Junqi Zhao 2021-11-05 08:07:34 UTC
Created attachment 1840034 [details]
400 Bad Request error

Description of problem:
login dev console with cluster-admin user, select one project, example, openshift-apiserver, click Observe, there are 4 dashboards under the Dashboard list

select "Kubernetes / Compute Resources / Namespace (Workloads)" dashboard, click the "inspect" button for "Receive Bandwidth", it leads to the "Metrics" tab, click the "Show PromQL" button to see the prometheus expression, query string is "sum(irate(container_network_receive_bytes_total{cluster="", namespace=~"openshift-apiserver"}[5m])) by (pod)", but it shows "Bad Request" for the result. 

Debugged in console, the response is "label matcher value (namespace="openshift-apiserver") conflicts with injected value (namespace=~"openshift-apiserver")", see from the picture.

there are result from prometheus
sum(irate(container_network_receive_bytes_total{cluster="", namespace=~"openshift-apiserver"}[5m])) by (pod)
{pod="apiserver-7d5d979bd6-kjsq4"}  6713.866666666667
{pod="apiserver-7d5d979bd6-jqz87"}  6250.733333333334
{pod="apiserver-7d5d979bd6-n7hzz"}  6350.633333333333

NOTE: checked in 4.9, it is the same expression in 4.9, but 4.9 does not have such issue.

# oc -n openshift-apiserver get pod
NAME                         READY   STATUS    RESTARTS   AGE
apiserver-7d5d979bd6-jqz87   2/2     Running   0          7h54m
apiserver-7d5d979bd6-kjsq4   2/2     Running   0          7h53m
apiserver-7d5d979bd6-n7hzz   2/2     Running   0          7h56m

if we change the expr to
sum(irate(container_network_receive_bytes_total{cluster="", namespace="openshift-apiserver"}[5m])) by (pod)
we can get correct result

other dashboard are also effected
Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2021-11-04-001635

How reproducible:
always

Steps to Reproduce:
1. see the descripiton
2.
3.

Actual results:
400 Bad Request error

Expected results:
no error

Additional info:

Comment 2 Junqi Zhao 2021-11-05 08:09:07 UTC
Comment on attachment 1840034 [details]
400 Bad Request error

expression is
sum(irate(container_network_receive_bytes_total{cluster="", namespace=~"openshift-apiserver"}[5m])) by (pod)

response is
label matcher value (namespace="openshift-apiserver") conflicts with injected value (namespace=~"openshift-apiserver")

Comment 3 Sahil Budhwar 2021-12-01 09:49:25 UTC
This bug is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=2024199 and can be closed. There is a PR linked in #2024199 which is merged and waiting on the operator version release.

Comment 4 Sahil Budhwar 2021-12-01 10:00:46 UTC

*** This bug has been marked as a duplicate of bug 2024199 ***