Bug 1458186 – Hawkular metrics rest api responding sporadically

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1458186 - Hawkular metrics rest api responding sporadically [NEEDINFO]

Summary:	Hawkular metrics rest api responding sporadically

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Hawkular (Show other bugs)
Sub Component:
Version:	3.4.1
Hardware:	Unspecified Unspecified

Priority	urgent Severity urgent
Target Milestone:	---
Target Release:	3.5.z
Assigned To:	John Sanda
QA Contact:	Junqi Zhao
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:	1500644
Blocks:
	Show dependency tree / graph

Reported:	2017-06-02 05:07 EDT by Miheer Salunke
Modified:	2017-12-07 02:10 EST (History)
CC List:	13 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: Extra, unnecessary queries were being performed on each request. The issue was logged upstream as https://issues.jboss.org/browse/HWKMETRICS-711. Consequence: The GET /hawkular/metrics/metrics endpoint could fail with timeouts. Fix: Only perform the extra queries when explicitly requested. By default, do not execute the extra queries which provide optional data. Result: The endpoint is more stable and not as susceptible to timeouts.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-12-07 02:10:26 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Flags:	misalunk: needinfo? (snegrea)

Attachments	(Terms of Use)
script to curl rest api (913 bytes, text/plain) 2017-10-26 05:37 EDT, Junqi Zhao	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
JBoss Issue Tracker	HWKMETRICS-711	Critical	Closed	MinMaxTimestamp of definitions should be fetched only on request	2018-10-25 20:51 EDT
Red Hat Product Errata	RHSA-2017:3389	normal	SHIPPED_LIVE	Moderate: Red Hat OpenShift Enterprise security, bug fix, and enhancement update	2017-12-07 07:09:10 EST

Groups: None (edit)

Comment 12 Matt Wringe 2017-06-20 14:07:26 EDT

Setting this to OCP component instead of Online.

Comment 17 Kenjiro Nakayama 2017-06-26 12:45:31 EDT

@Stefan, as you requested we will collect the logs again, but could you please give us your comment about below error, which is outputting the customer's env?

~~~
  E0508 06:26:09.276956       1 client.go:243] Post https://hawkular-metrics:443/hawkular/metrics/counters/data: dial tcp 172.30.xxx.xx:443: getsockopt: no route to host
~~~

Comment 18 Michael Burman 2017-06-29 06:44:00 EDT

getsockopt is a sign that Kubernetes networking can't communicate between pods. These requests will never reach Hawkular-Metrics (the requests stop in the networking layer already).

Comment 19 Kenjiro Nakayama 2017-06-29 07:10:43 EDT

Yeah, so I think that op team or someone who can access to the cluster should investigate the network more. But we should collect new metrics logs? (I am asking because ops let us file this ticket and ask dev team.)

Comment 24 John Sanda 2017-07-31 11:57:40 EDT

This issue was originally fixed upstream in HWKMETRICS-625. We will back port it in HWKMETRICS-711 so that it can go into OCP 3.5.1.

Comment 25 Bernie Hoefer 2017-09-11 08:37:03 EDT

From an e-mail on 2017-08-28, this fix was waiting on Hawkular Metrics 0.23.10.  Is that still the case, please?  Thanks.

Comment 26 Matt Wringe 2017-09-11 09:55:09 EDT

It looks like in OCP 3.5.1 we are using 0.23.8, so the update has not yet been applied.

Comment 28 Bernie Hoefer 2017-09-25 08:28:29 EDT

(In reply to Matt Wringe from comment #26)
===
> It looks like in OCP 3.5.1 we are using 0.23.8, so the update has not yet
> been applied.
===

Is this still the case?

Comment 31 Miheer Salunke 2017-10-24 00:46:55 EDT

Can you please share the errata for this ?

Comment 32 Junqi Zhao 2017-10-26 05:35:14 EDT

created pod within projects and let metrics running for 8 hours, during this time
use oc client to run 
#for i in {0..99}; ./shell.sh ; done

shell.sh is script to curl metrics, see the attached file.

checked metrics log, there did not have exceptions in pods log and rest api was worked well.


# openshift version
openshift v3.5.5.31.39
kubernetes v1.5.2+43a9be4
etcd 3.1.0


metrics-hawkular-metrics/images/3.5.0-50
metrics-cassandra/images/3.5.0-41
metrics-heapster/images/3.5.0-33

Comment 33 Junqi Zhao 2017-10-26 05:37 EDT

Created attachment 1343645 [details]
script to curl rest api

Comment 36 errata-xmlrpc 2017-12-07 02:10:26 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389

Note You need to log in before you can comment on or make changes to this bug.