Bug 1588420

Summary: SDN related metrics cannot be captured by prometheus
Product: OpenShift Container Platform Reporter: Meng Bo <bmeng>
Component: NetworkingAssignee: jtanenba
Status: CLOSED ERRATA QA Contact: Meng Bo <bmeng>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.10.0CC: anli, anusaxen, aos-bugs, bbennett, cdc, dcbw, hongli, jforrest, pasik, spasquie, zzhao
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:40:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Meng Bo 2018-06-07 09:57:10 UTC
Description of problem:
When setup env with prometheus enabled. The networking related metrics cannot be found in prometheus console.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Setup ocp cluster with prometheus enabled
2. Login to the prometheus console to check the sdn related metrics 

Actual results:
Cannot find the openshift sdn related items

Expected results:
Should be able to capture the networking related metrics by prometheus.

Additional info:

Comment 1 Meng Bo 2018-06-07 10:01:21 UTC
The following metrics should be shown in prometheus console. 


Comment 2 Ben Bennett 2018-06-07 13:04:23 UTC
This is presumably not a regression so moving to 3.11.

Reassigning to the metrics team since I assume they are in charge of hooking up metrics.

Comment 3 Frederic Branczyk 2018-06-12 13:34:56 UTC
Correct, this is not a regression, it's simply not configured. Assigning this to Casey for the SDN team to configure metrics collection for 3.11.

Comment 4 Casey Callendrello 2018-06-12 13:40:07 UTC
Where did the list of metrics in comment 1 come from? Were these metrics previously exposed via hawkular?

Comment 5 Meng Bo 2018-06-20 08:05:05 UTC
(In reply to Casey Callendrello from comment #4)
> Where did the list of metrics in comment 1 come from? Were these metrics
> previously exposed via hawkular?

Here is a list which were defined for networking related metrics, and I designed test case based on this

I am not sure if they work with hawkular, we do not have case about networking metrics via hawkular.

Comment 6 Casey Callendrello 2018-06-20 09:25:09 UTC
Great, thanks for the info.

Assigning to dcbw, who wrote that code.

Comment 7 Jessica Forrester 2018-08-02 20:05:24 UTC
Since this is specifically monitoring of the networking stack I'm moving this to their BZ component for tracking.

Comment 8 Meng Bo 2018-08-07 09:29:29 UTC
Any progress on this? I can still see this problem on v3.11.0

Comment 9 Dan Williams 2018-08-07 21:05:16 UTC
Casey, I'd thought I did everything needed to expose those, but now that the SDN is a daemonset perhaps more is required?  Any idea what's needed to actually push the metrics out now?

Comment 10 Casey Callendrello 2018-08-09 13:55:18 UTC
The question is: where's the problem? Is the daemonst correctly answering on the metrics endpoint? Is Prometheus configured to scrape that endpoint?

Should be easy enough to answer the first question. I'll also ask the monitoring people if they can answer the second.

Comment 11 Meng Bo 2018-08-10 02:46:20 UTC
The prometheus related services are running in the openshift-metrics project, and the sdn related services are running in the openshift-sdn project. 

Is it able to read the info in openshift-sdn from openshift-metrics?

Comment 12 Casey Callendrello 2018-08-14 16:01:59 UTC
Figured this one out.

We need to:
1) Decide on a port for the sdn to use
2) Add that to "metrics-bind-address" in ProxyArguments
3) Configure a headless service for the metrics port with appropriate labels (see etcd for an example)
4) Create a ServiceMonitor object
5) Profit!

Comment 13 Casey Callendrello 2018-08-16 14:09:50 UTC
Clayton says he'll "take care of this..."

Comment 14 zhaozhanqi 2019-01-25 06:56:27 UTC
hi, this issue still has not fixed in 4.0, payload 4.0.0-0.nightly-2019-01-24-184525

since the target release the 4.0.0. So any progress on this?

Comment 15 Casey Callendrello 2019-01-25 16:14:53 UTC
Yes, Jacob has been making progress and this should be in soon.

Comment 16 jtanenba 2019-02-13 12:43:51 UTC
Posted and has been merged 


Comment 19 Anurag saxena 2019-03-29 20:00:39 UTC
Verified it on 4.0.0-0.nightly-2019-03-28-210640. Observed following SDN related metrics are now captured on Prometheus console:



Comment 21 errata-xmlrpc 2019-06-04 10:40:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.