Bug 1825207

Summary: [4.5] 504 error for Prometheus API in administrator console with kuryr network
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: NetworkingAssignee: Maysa Macedo <mdemaced>
Networking sub component: kuryr QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: juriarte, mdemaced, rlobillo
Version: 4.5Keywords: AutomationBackLog, TestBlocker
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1825215 (view as bug list) Environment:
Last Closed: 2020-07-13 17:28:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1825215    
Attachments:
Description Flags
504 error for the Prometheus APIs for "Home -> Overview" page
none
failed to load metrics list due to 504 error
none
GET requests from console GUI to prometheus API
none
"Monitoring -> Metrics" menu from console GUI
none
kube_pod_info_node_coun metric retrieval from console GUI none

Description Junqi Zhao 2020-04-17 11:54:53 UTC 
Created attachment 1679633 [details]
504 error for the Prometheus APIs for "Home -> Overview" page

Created attachment 1679633 [details]
504 error for the Prometheus APIs for "Home -> Overview" page

Description of problem:
This bug is from Bug 1822861, use it to track the 504 error for the Prometheus APIs in the administrator console.

Login the administrator console, check in "Home -> Overview" page, "Monitoring -> Alerting", or "Monitoring -> Metrics" page, the Prometheus APIs in the administrator console returns 504 error in the kuryr OCP 4.5.0-0.nightly-2020-04-17-053057 cluster, see the attached picture 



Version-Release number of selected component (if applicable):
kuryr OCP 4.5.0-0.nightly-2020-04-17-05305

How reproducible:
always

Steps to Reproduce:
1. See the description
2.
3.

Actual results:
504 error for Prometheus API in administrator console 

Expected results:
no error

Additional info:
Comment 1 Junqi Zhao 2020-04-17 12:03:22 UTC 
Created attachment 1679637 [details]
failed to load metrics list due to 504 error
Comment 2 Jon Uriarte 2020-04-30 13:47:13 UTC 
Verified in 4.5.0-0.nightly-2020-04-29-144201 on top of RHOS_TRUNK-16.0-RHEL-8-20200427.n.0 OSP 16 compose.
OSP 16 deployment with OVS neutron backend.

OCP 4.5 installation with Kuryr completed successfully.

There are no 504 error responses to Prometheus metrics requests in the GUI "Home -> Overview" page.
The metrics are retrieved correctly in "Monitoring -> Metrics"

The metrics can be retrieved from the CLI as well:

$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.196.2.22:9100/metrics'
Comment 3 Jon Uriarte 2020-04-30 13:48:33 UTC 
Created attachment 1683314 [details]
GET requests from console GUI to prometheus API
Comment 4 Jon Uriarte 2020-04-30 13:49:25 UTC 
Created attachment 1683315 [details]
"Monitoring -> Metrics" menu from console GUI
Comment 5 Jon Uriarte 2020-04-30 13:50:11 UTC 
Created attachment 1683316 [details]
kube_pod_info_node_coun metric retrieval from console GUI
Comment 7 errata-xmlrpc 2020-07-13 17:28:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409