Bug 1361061
| Summary: | Failed to show metrics on web console due to the clocks between machine running the console and the machine running Hawkular Metrics are out of sync | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | chunchen <chunchen> | ||||||
| Component: | Hawkular | Assignee: | Samuel Padgett <spadgett> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | chunchen <chunchen> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 3.2.1 | CC: | aos-bugs, jforrest, mwringe, tdawson, wmeng, wsun, xiazhao | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Cause:
The web console previously used the client's clock to calculate the start time for displaying metrics.
Consequence:
If the client's clock was more than one hour faster than the server clock, an occur would occur opening the metrics tab in the web console.
Fix:
The web console now uses the server time for calculating start and end times for metrics.
Result:
Metrics will display properly even if the client clock is out of sync with the server.
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2016-09-27 09:41:55 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Where is this error message showing up exactly? In one of the metric component logs? In the browser's console logs? There are also no screenshots attached like mentioned in the bz. Created attachment 1186221 [details]
console screenshot 2
Created attachment 1186222 [details]
console screenshot 1
Sorry for forgetting to add the attachments, they are existed now. Ah, thank you. I think I know the problem now. The browser is using its own clock to send the 'start' time in that response, and the end time is not specified. When the end value is not specified, it means that Hawkular Metrics will use its own system clock to generate this value. What is most likely happening here is that the clocks between the machine running the console and the machine running Hawkular Metrics are out of sync and the end value (generated by Hawkular Metrics) is before the start value (generated by the console). Can you please verify if the clocks between these system are indeed off by a large margin? The upstream issue for this is https://issues.jboss.org/browse/HWKMETRICS-358 Once that is done, the console can use relative timestamps and this type of issue cannot occur. Yes, The issue can be reproduced when the clocks are out of sync between these system even if I tested with metrics in V1 registry(brew-pulp-docker01.web.prod...com:8888), so it's not related to V2 registry, I am removing the keywords in the bug title. For this root cause, the bug is not a testblocker any more. The metrics containers now support relative timestamps. The only other piece is for the console in OSE 3.3 to be updated to use it. Its already working in Origin with relative timestamps. This has been merged into ose and is in OSE v3.3.0.22 or newer. It's fixed, checked with the latest hawkular 3.3.0 image(70c30be69f7d), so mark it as verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1933 |
Description of problem: It's failed to show metrics on web console, the page response: '{"errorMsg":"Range end must be strictly greater than start"}' Version-Release number of selected component (if applicable): openshift v3.2.1.4-1-g1864c8f kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 brew-pulp-docker01.web.qa...com:8888/openshift3/metrics-hawkular-metrics 3.2.1 219e26f45297 brew-pulp-docker01.web.qa...com:8888/openshift3/metrics-cassandra 3.2.1 afeae5fccd3f brew-pulp-docker01.web.qa...com:8888/openshift3/metrics-heapster 3.2.1 eac7eb4e46c4 How reproducible: Always Steps to Reproduce: 1. Login to OpenShift server and use openshift-infra project oc project openshift-infra 2. Deploy metrics stack oc create serviceaccount metrics-deployer oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-infra:heapster oc policy add-role-to-user edit system:serviceaccount:openshift-infra:metrics-deployer oc secrets new metrics-deployer nothing=/dev/null oc new-app metrics-deployer-template -p IMAGE_PREFIX=brew-pulp-docker01.web.qa...com:8888/openshift3/,IMAGE_VERSION=3.2.1,CASSANDRA_PV_SIZE=10,CASSANDRA_NODES=1,MASTER_URL=https://openshift-221...com:8443,HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.0725-6cj.qe.rhcloud.com,USE_PERSISTENT_STORAGE=false,MODE=deploy 3. Check the pods oc get pods 4. Try to show the pod metrics on web console Actual results: at step 3: NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-z42ez 1/1 Running 0 1h hawkular-metrics-8tzy6 1/1 Running 16 1h heapster-p6oe0 1/1 Running 14 1h metrics-deployer-437pk 0/1 Completed 0 1h at step 4: the metrics can not be shown on console, please refer to the web console screenshot in attachments. Expected results: Should show metrcs on web console Additional info: