Bug 1703777

Summary: sometimes cluster console is not showing projects list when user logged in
Product: OpenShift Container Platform Reporter: Yadan Pei <yapei>
Component: Management ConsoleAssignee: Samuel Padgett <spadgett>
Status: CLOSED ERRATA QA Contact: Yadan Pei <yapei>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, jokerman, mmccomas, yapei
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
In rare cases, a race condition could cause the project list to fail to load after logging into the admin console. The user would need to refresh the page to see the list of projects. The problem has been fixed, and projects now load successfully after login.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-24 08:08:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
DevConsole Error
none
HAR none

Description Yadan Pei 2019-04-28 09:25:45 UTC
Description of problem:
Sometimes when we open admin console , it keeps loading and return nothing until we click some other menus or reload button.
Usually it happens on the first time user opens admin console and only FF in my trying(not sure if it only reproduces on FF)

Version-Release number of selected component (if applicable):
v3.11.98

How reproducible:
Sometimes

Steps to Reproduce:
1. user open admin console URL
2.
3.

Actual results:
1. it keeps loading and nothing returned after a long time, in developer console we can see
Error: "Forbidden"
    f index.tsx:52
    v index.tsx:52

Expected results:
1. project list page should be returned with projects or empty 

Additional info:

Comment 1 Yadan Pei 2019-04-28 09:26:15 UTC
Created attachment 1559610 [details]
DevConsole Error

Comment 2 Samuel Padgett 2019-04-28 12:56:31 UTC
> Actual results:
> 1. it keeps loading and nothing returned after a long time, in developer
> console we can see
> Error: "Forbidden"
>     f index.tsx:52
>     v index.tsx:52

That error is expected depending on your permissions and should not cause a problem.

Any chance you can record a video of what's happening, ideally with the network tab open? It's not clear to me exactly what you're seeing. You keep seeing the loading indicator on the projects page? Is it possible the API response is simply slow? I would check that the initial request for projects has completed in the network tab.(In reply to Yadan Pei from comment #0)

Comment 3 Yadan Pei 2019-04-29 06:34:04 UTC
Sure, I will add a screenshot next time I meet this issue. Yes, only a loading indicator on projects page, I met this when I logins as a normal user and I have only two projects.

Comment 10 Yadan Pei 2019-05-07 07:08:00 UTC
Sam, thanks for your trying. I didn't reproduce today in 3.11 cluster, I will give necessary files next time I meet this issue.

Comment 13 Yadan Pei 2019-05-31 05:31:18 UTC
I tried to reproduce today but without luck. I will keep trying in following days to see if CLI works when this issue is reproduced.

Comment 14 Samuel Padgett 2019-08-07 20:37:23 UTC
Hi, have you been able to reproduce?

Comment 15 Yadan Pei 2019-08-08 03:24:46 UTC
Hi Sam,

I didn't reproduce recently since we did little testing on 3.11, feel free to close it if you want.

Comment 17 Yadan Pei 2019-08-08 05:25:56 UTC
Using oc login can successfully login

Comment 19 Yadan Pei 2019-08-09 08:58:50 UTC
It happens mostly when switch from Developer Console to Cluster Console, visiting Cluster Console route directly seems can't reproduce the issue

Attached is HAR file

Comment 20 Yadan Pei 2019-08-09 08:59:26 UTC
Created attachment 1602071 [details]
HAR

Comment 23 Samuel Padgett 2019-09-09 19:54:10 UTC
It looks like the request for `openshift-favicon.png` set a new `csrf-token` cookie while other requests were being fired off. There is a race condition where GET requests read the current cookie and add the header to the next request. If another response updates the cookie between the time it's read and when the request is sent, the values don't match. I'm not sure why a new cookie was be set for the favicon request, however.

This should be fixed indirectly by https://github.com/openshift/console/pull/2523, but I'd like to understand why the CSRF token was updated.

Comment 24 Samuel Padgett 2019-09-09 20:04:16 UTC
The favicon path is wrong: /auth/static/assets/openshift-favicon.png

This caused the index handler to run, which reset the CSRF cookie.

https://github.com/openshift/console/blob/master/pkg/server/server.go#L318

Comment 25 Samuel Padgett 2019-09-09 20:07:36 UTC
The paths in tokener.html are wrong:

https://github.com/openshift/console/blob/release-3.11/frontend/public/tokener.html

This is only a problem in 3.11.

Comment 28 Yadan Pei 2019-09-18 08:02:40 UTC
# oc get pods -n openshift-console -o yaml | grep image
      image: registry.reg-aws.openshift.com:443/openshift3/ose-console:v3.11
# docker inspect registry.reg-aws.openshift.com:443/openshift3/ose-console:v3.11
....
"Labels":{
"License":"GPLv2+",
"architecture":"x86_64",
"authoritative-source-url":"registry.access.redhat.com",
"build-date":"2019-09-17T22:08:49.979227",
"com.redhat.build-host":"cpt-1004.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component":"openshift-enterprise-console-container",
"com.redhat.license_terms":"https://www.redhat.com/en/about/red-hat-end-user-license-agreements",
"description":"ThisisacomponentofOpenShiftContainerPlatformandprovidesawebconsole.",
"distribution-scope":"public",
"io.k8s.description":"ThisisacomponentofOpenShiftContainerPlatformandprovidesawebconsole.",
"io.k8s.display-name":"OpenShiftConsole",
"io.openshift.build.commit.id":"2abfa57365be06e6e14e751ef4b4a6a0dfc8fdaa",
"io.openshift.build.commit.url":"https://github.com/openshift/console/commit/2abfa57365be06e6e14e751ef4b4a6a0dfc8fdaa",
"io.openshift.build.source-location":"https://github.com/openshift/console",
"io.openshift.tags":"openshift,console",
"maintainer":"SamuelPadgett<spadgett>",
"name":"openshift3/ose-console",
"release":"1",
"summary":"ProvidesthelatestreleaseofRedHatEnterpriseLinux7inafullyfeaturedandsupportedbaseimage.",
"url":"https://access.redhat.com/containers/#/registry.access.redhat.com/openshift3/ose-console/images/v3.11.146-1",
"vcs-ref":"8df6ebf6dfbd0a53196640d0e3b60006af4bcb57",
"vcs-type":"git",
"vendor":"RedHat,Inc.",
"version":"v3.11.146"
....

I don't see this issue happen on 3.11.146

Comment 29 Yadan Pei 2019-09-18 08:03:10 UTC
Thanks very much for the fix and will continue to track this issue in next few builds

Comment 31 errata-xmlrpc 2019-09-24 08:08:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2816