Bug 1395111

Summary: openshfit metrics need support historical metrics feature
Product: OpenShift Container Platform Reporter: DeShuai Ma <dma>
Component: RFEAssignee: Dan McPherson <dmcphers>
Status: CLOSED WONTFIX QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.4.0CC: aos-bugs, decarr, jokerman, mmccomas, sross
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-17 15:52:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description DeShuai Ma 2016-11-15 08:09:24 UTC 
Description of problem:
The historical metrics feature[1] has completed in upstream for a long time. But this can't support Hawkular.
As our origin-metrics based on hawkular+cassandra, so we can't use this feature in our ocp-3.4.
The latest test result for this feature in openshift metrics is always failed with error[2].
[1]https://trello.com/c/6WsSmMcx/340-3-heapster-should-provide-an-interface-to-retrieve-historical-metrics-oldtimer
[2]http://pastebin.test.redhat.com/427419

Version-Release number of selected component (if applicable):
Always

How reproducible:
openshift v3.4.0.25+1f36858
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

Steps to Reproduce:
1.Enable historical metrics in hepaster for openshfit-metrics
[root@ip-172-18-1-230 ~]# oc logs heapster-1o8bv
Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status
The endpoint check has successfully completed.
Starting Heapster with the following arguments: --source=kubernetes:https://ip-172-18-1-230.ec2.internal:8443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=-M0qR7Q1Cj6rX4n&filter=label(container_name:^system.slice.*|^user.slice) --historical_source=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=-M0qR7Q1Cj6rX4n&filter=label(container_name:^system.slice.*|^user.slice) --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=15s
E1107 03:44:26.360139       1 factory.go:85] Sink type "hawkular" does not support being used for historical access
E1107 03:44:26.361290       1 factory.go:102] Error while initializing historical access: unable to use sink "hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=-M0qR7Q1Cj6rX4n&filter=label(container_name:^system.slice.*|^user.slice)" as a historical source
F1107 03:44:26.361441       1 heapster.go:96] Failed to use a sink as a historical metrics source
goroutine 1 [running]:
k8s.io/heapster/vendor/github.com/golang/glog.stacks(0x3556f00, 0x0, 0x0, 0x0)
        /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:766 +0xb8
k8s.io/heapster/vendor/github.com/golang/glog.(*loggingT).output(0x3536c00, 0xc800000003, 0xc8200be300, 0x35022af, 0xb, 0x60, 0x0)
        /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:717 +0x259
k8s.io/heapster/vendor/github.com/golang/glog.(*loggingT).printDepth(0x3536c00, 0x3, 0x1, 0xc820523d50, 0x1, 0x1)
        /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:646 +0x1b8
k8s.io/heapster/vendor/github.com/golang/glog.(*loggingT).print(0x3536c00, 0x3, 0xc820523d50, 0x1, 0x1)
        /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:637 +0x50
k8s.io/heapster/vendor/github.com/golang/glog.Fatal(0xc820523d50, 0x1, 0x1)
        /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:1125 +0x49
main.main()
        /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/metrics/heapster.go:96 +0xa46

2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 DeShuai Ma 2016-11-15 08:13:41 UTC 
Use this bug to trace how ocp support historical metrics feature. I am not sure if we want this feature in 3.4 or not. need confirm
Comment 2 Derek Carr 2016-11-15 21:35:38 UTC 
This is not supported yet on Hawkular.
Comment 3 DeShuai Ma 2016-11-16 02:16:29 UTC 
I‘d like change the bug to a RFE, if we close this directly then we will forget it.
Comment 4 Dan McPherson 2016-11-16 12:20:29 UTC 
What exactly would we get out of this?  Hawkular already keeps historical metrics.  Also is this appropriate to do given heapster's future?
Comment 5 Solly Ross 2016-11-17 15:47:47 UTC 
this is no longer necessary.  Eventually, we will get a historical API definition as part of the new monitoring pipeline, so we'll need to implement that, but getting the Heapster historical API support is not necessary in light of new monitoring vision.