Bug 1458186 - Hawkular metrics rest api responding sporadically
Summary: Hawkular metrics rest api responding sporadically
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.5.z
Assignee: John Sanda
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1500644
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-02 09:07 UTC by Miheer Salunke
Modified: 2023-09-15 00:02 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Extra, unnecessary queries were being performed on each request. The issue was logged upstream as https://issues.jboss.org/browse/HWKMETRICS-711. Consequence: The GET /hawkular/metrics/metrics endpoint could fail with timeouts. Fix: Only perform the extra queries when explicitly requested. By default, do not execute the extra queries which provide optional data. Result: The endpoint is more stable and not as susceptible to timeouts.
Clone Of:
Environment:
Last Closed: 2017-12-07 07:10:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
script to curl rest api (913 bytes, text/plain)
2017-10-26 09:37 UTC, Junqi Zhao 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker HWKMETRICS-711 0 Critical Closed MinMaxTimestamp of definitions should be fetched only on request 2020-07-29 06:56:37 UTC
Red Hat Product Errata RHSA-2017:3389 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Enterprise security, bug fix, and enhancement update 2017-12-07 12:09:10 UTC

Comment 12 Matt Wringe 2017-06-20 18:07:26 UTC 
Setting this to OCP component instead of Online.
Comment 17 Kenjiro Nakayama 2017-06-26 16:45:31 UTC 
@Stefan, as you requested we will collect the logs again, but could you please give us your comment about below error, which is outputting the customer's env?

~~~
  E0508 06:26:09.276956       1 client.go:243] Post https://hawkular-metrics:443/hawkular/metrics/counters/data: dial tcp 172.30.xxx.xx:443: getsockopt: no route to host
~~~
Comment 18 Michael Burman 2017-06-29 10:44:00 UTC 
getsockopt is a sign that Kubernetes networking can't communicate between pods. These requests will never reach Hawkular-Metrics (the requests stop in the networking layer already).
Comment 19 Kenjiro Nakayama 2017-06-29 11:10:43 UTC 
Yeah, so I think that op team or someone who can access to the cluster should investigate the network more. But we should collect new metrics logs? (I am asking because ops let us file this ticket and ask dev team.)
Comment 24 John Sanda 2017-07-31 15:57:40 UTC 
This issue was originally fixed upstream in HWKMETRICS-625. We will back port it in HWKMETRICS-711 so that it can go into OCP 3.5.1.
Comment 25 Bernie Hoefer 2017-09-11 12:37:03 UTC 
From an e-mail on 2017-08-28, this fix was waiting on Hawkular Metrics 0.23.10.  Is that still the case, please?  Thanks.
Comment 26 Matt Wringe 2017-09-11 13:55:09 UTC 
It looks like in OCP 3.5.1 we are using 0.23.8, so the update has not yet been applied.
Comment 28 Bernie Hoefer 2017-09-25 12:28:29 UTC 
(In reply to Matt Wringe from comment #26)
===
> It looks like in OCP 3.5.1 we are using 0.23.8, so the update has not yet
> been applied.
===

Is this still the case?
Comment 31 Miheer Salunke 2017-10-24 04:46:55 UTC 
Can you please share the errata for this ?
Comment 32 Junqi Zhao 2017-10-26 09:35:14 UTC 
created pod within projects and let metrics running for 8 hours, during this time
use oc client to run 
#for i in {0..99}; ./shell.sh ; done

shell.sh is script to curl metrics, see the attached file.

checked metrics log, there did not have exceptions in pods log and rest api was worked well.


# openshift version
openshift v3.5.5.31.39
kubernetes v1.5.2+43a9be4
etcd 3.1.0


metrics-hawkular-metrics/images/3.5.0-50
metrics-cassandra/images/3.5.0-41
metrics-heapster/images/3.5.0-33
Comment 33 Junqi Zhao 2017-10-26 09:37:33 UTC 
Created attachment 1343645 [details]
script to curl rest api
Comment 36 errata-xmlrpc 2017-12-07 07:10:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389
Comment 37 Red Hat Bugzilla 2023-09-15 00:02:25 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days
Note You need to log in before you can comment on or make changes to this bug.