Bug 1283714 - After leaving pod metrics tab open for a long time, 403/forbidden errors occur
After leaving pod metrics tab open for a long time, 403/forbidden errors occur
Status: CLOSED WORKSFORME
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metrics (Show other bugs)
3.1.0
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Matt Wringe
chunchen
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-19 11:26 EST by Erik M Jacobs
Modified: 2016-09-29 22:17 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-16 10:31:26 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Erik M Jacobs 2015-11-19 11:26:06 EST
rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-cassandra           latest              8ea21f4b3377        4 weeks ago         472.7 MB
rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-deployer            latest              277d6f10f28e        8 days ago          506.7 MB
rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-hawkular-metrics    latest              b44dc66d64f2        2 weeks ago         1.109 GB
rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-heapster            latest              800434e62203        4 weeks ago         228.4 MB

atomic-openshift-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-clients-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-clients-redistributable-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-master-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-node-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-pod-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-sdn-ovs-3.1.0.4-1.git.4.b6c7cd2.el7aos.x86_64
atomic-openshift-utils-3.0.13-1.git.0.5e8c5c7.el7aos.noarch

If you visit a pod details page and go to the metrics tab and leave it open for more than fifteen minutes, you end up with some 403 errors:

https://metrics.cloudapps.example.com/hawkular/metrics/counters/postgresql%…a-11e5-abc2-525400b33d1d%2Fcpu%2Fusage/data?buckets=61&start=1447943224976 Failed to load resource: the server responded with a status of 403 (Forbidden)

There was a thought that this was related to token expiry, but I can switch to a different page/tab without having to reload, or change the metrics timeframe (eg: 30 minutes to 1 hour) and have no issues.
Comment 1 Matt Wringe 2015-11-19 11:31:50 EST
This still really sounds like a token expiry issue. 

I suspect that when switching tabs or changing the timeline, its updating the token its using.

@spadgett: do you know how tokens are being handled here?
Comment 2 Samuel Padgett 2015-11-19 11:58:01 EST
We're not changing tokens when navigating tabs. I wouldn't expect the token to expire after 15 minutes, and if it does, it should expire everywhere. We're passing the same token to Hawkular Metrics that we pass to the API server.
Comment 3 Matt Wringe 2015-11-19 12:23:08 EST
Ok, thanks for the info Sam. I will debug it on my end and see what is going on.
Comment 4 Yanping Zhang 2015-11-19 22:52:50 EST
Check on chrome47, keep metric tab open for about half an hour, can not meet the 403 error. Is there other condition to reproduce this issue?
Comment 5 Matt Wringe 2015-12-02 09:07:32 EST
@Erik : we are still waiting for the answer to Yanping's question. Is there anything else special in your setup or something else you have changed with your OpenShift setup? Or is this just a simple out of the box install of OpenShift?
Comment 6 Erik M Jacobs 2015-12-02 09:23:59 EST
My setup is this (in fact this doc was built from my setup):

https://github.com/openshift/training/blob/master/01-Requirements-and-Preparation.md

3 KVM VMs running on my laptop using ose3-(master/node1/node2).example.com as hostnames. Everything is vanilla. Nothing special. I'm even using the non-advanced installer.

I haven't reinstalled lately with GA so it might be worth another shot to see if it happens.
Comment 7 Matt Wringe 2016-01-11 13:22:34 EST
@Erik: can you confirm that you are still seeing this issue?

Note You need to log in before you can comment on or make changes to this bug.