Hide Forgot
Description of problem: The historical metrics feature[1] has completed in upstream for a long time. But this can't support Hawkular. As our origin-metrics based on hawkular+cassandra, so we can't use this feature in our ocp-3.4. The latest test result for this feature in openshift metrics is always failed with error[2]. [1]https://trello.com/c/6WsSmMcx/340-3-heapster-should-provide-an-interface-to-retrieve-historical-metrics-oldtimer [2]http://pastebin.test.redhat.com/427419 Version-Release number of selected component (if applicable): Always How reproducible: openshift v3.4.0.25+1f36858 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 Steps to Reproduce: 1.Enable historical metrics in hepaster for openshfit-metrics [root@ip-172-18-1-230 ~]# oc logs heapster-1o8bv Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status The endpoint check has successfully completed. Starting Heapster with the following arguments: --source=kubernetes:https://ip-172-18-1-230.ec2.internal:8443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=-M0qR7Q1Cj6rX4n&filter=label(container_name:^system.slice.*|^user.slice) --historical_source=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=-M0qR7Q1Cj6rX4n&filter=label(container_name:^system.slice.*|^user.slice) --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=15s E1107 03:44:26.360139 1 factory.go:85] Sink type "hawkular" does not support being used for historical access E1107 03:44:26.361290 1 factory.go:102] Error while initializing historical access: unable to use sink "hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=-M0qR7Q1Cj6rX4n&filter=label(container_name:^system.slice.*|^user.slice)" as a historical source F1107 03:44:26.361441 1 heapster.go:96] Failed to use a sink as a historical metrics source goroutine 1 [running]: k8s.io/heapster/vendor/github.com/golang/glog.stacks(0x3556f00, 0x0, 0x0, 0x0) /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:766 +0xb8 k8s.io/heapster/vendor/github.com/golang/glog.(*loggingT).output(0x3536c00, 0xc800000003, 0xc8200be300, 0x35022af, 0xb, 0x60, 0x0) /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:717 +0x259 k8s.io/heapster/vendor/github.com/golang/glog.(*loggingT).printDepth(0x3536c00, 0x3, 0x1, 0xc820523d50, 0x1, 0x1) /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:646 +0x1b8 k8s.io/heapster/vendor/github.com/golang/glog.(*loggingT).print(0x3536c00, 0x3, 0xc820523d50, 0x1, 0x1) /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:637 +0x50 k8s.io/heapster/vendor/github.com/golang/glog.Fatal(0xc820523d50, 0x1, 0x1) /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/vendor/github.com/golang/glog/glog.go:1125 +0x49 main.main() /builddir/build/BUILD/heapster-1.2.0/_build/src/k8s.io/heapster/metrics/heapster.go:96 +0xa46 2. 3. Actual results: Expected results: Additional info:
Use this bug to trace how ocp support historical metrics feature. I am not sure if we want this feature in 3.4 or not. need confirm
This is not supported yet on Hawkular.
I‘d like change the bug to a RFE, if we close this directly then we will forget it.
What exactly would we get out of this? Hawkular already keeps historical metrics. Also is this appropriate to do given heapster's future?
this is no longer necessary. Eventually, we will get a historical API definition as part of the new monitoring pipeline, so we'll need to implement that, but getting the Heapster historical API support is not necessary in light of new monitoring vision.