Bug 1697803

Summary: Console always send request about api/kubernetes/apis/
Product: OpenShift Container Platform Reporter: shahan <hasha>
Component: Management ConsoleAssignee: Samuel Padgett <spadgett>
Status: CLOSED CURRENTRELEASE QA Contact: Yadan Pei <yapei>
Severity: medium Docs Contact:
Priority: high    
Version: 4.1.0CC: aos-bugs, jokerman, mmccomas, xiaocwan, yanpzhan, yapei
Target Milestone: ---Keywords: BetaBlocker
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-15 20:17:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description shahan 2019-04-09 06:59:07 UTC
Description of problem:
 Console always send requests to api/kubernetes/apis/, basically can reach 1000 requests every minute. This makes the console webpage particularly much slower than usual.
 From developer console, it shows following error:
  Failed to load resource: the server responded with a status of 503 (Service Unavailable)
operator-group.tsx:51 Error: Call to api/kubernetes/apis timed out after 20000ms.
    at new t (operator-group.tsx:51)
    at operator-group.tsx:51

$ oc api-resources --api-group=quota.openshift.io
NAME   SHORTNAMES   APIGROUP   NAMESPACED   KIND
error: unable to retrieve the complete list of server APIs: quota.openshift.io/v1: the server is currently unable to handle the request

Not sure frequently requested issue if it's caused by the unavailable service.
If so, the console should have a way to bypass effect influence from unavailable service


Version-Release number of selected component (if applicable):
Cluster version is 4.0.0-0.nightly-2019-04-05-165550


How reproducible:
Always

Steps to Reproduce:
1. as above described
2.
3.

Actual results:
 see attachments

Expected results:
 Should not always send the request to api/kubernetes/apis/

Additional info:

Comment 1 shahan 2019-04-09 06:59:31 UTC
console commits: 87cf25b49a98ef8d813c557ad4b74f0d18bc7d15

Comment 2 XiaochuanWang 2019-04-09 09:00:58 UTC
Effect more pages on same version:
When user created and subscribed an Etcd operator, check the "All Instance" page from Etcd operator detail page, it keeps requesting in loop. Please check the below pages:

All Instances (url: k8s/ns/openshift-operators/clusterserviceversions/etcdoperator.v0.9.4-clusterwide/instances)

etcd Cluster (url: k8s/ns/openshift-operators/clusterserviceversions/etcdoperator.v0.9.4-clusterwide/etcd.database.coreos.com~v1beta2~EtcdCluster)

etcd Backup (url: k8s/ns/openshift-operators/clusterserviceversions/etcdoperator.v0.9.4-clusterwide/etcd.database.coreos.com~v1beta2~EtcdBackup)

etcd Restore (url: k8s/ns/openshift-operators/clusterserviceversions/etcdoperator.v0.9.4-clusterwide/etcd.database.coreos.com~v1beta2~EtcdRestore)

Comment 3 Samuel Padgett 2019-04-09 11:26:38 UTC
This is caused by bug 1696387 where the quota API resource returns errors during discovery. We should change the console to tolerate it better, however.

Bug 1697225 is also a symptom of this bug because it triggers re-render of the secret page, losing state.

Comment 4 shahan 2019-04-10 09:25:52 UTC
Sam, any workaround for this issue? This console has been too slow to hard use. Hope the original issue could be fixed ASAP.

Comment 5 Samuel Padgett 2019-04-10 12:51:54 UTC
We have a basic fix in https://github.com/openshift/console/pull/1404. A workaround in the meantime is to log in with a normal user instead of kubeadmin. You shouldn't see the problem if you don't have authority to watch APIServices.

Comment 6 Samuel Padgett 2019-04-10 18:42:16 UTC
The problem should disappear entirely now that https://github.com/openshift/origin/pull/22425 has merged.

We're still working on making console more robust when an APIService is misbehaving.

Comment 7 Samuel Padgett 2019-04-11 12:41:50 UTC
Two fixes merged to address this. One that fixes the cause in master:

https://github.com/openshift/origin/pull/22425

And another that makes the console more tolerant of misbehaving API services:

https://github.com/openshift/console/pull/1404

Comment 9 shahan 2019-04-12 03:25:36 UTC
Now the console runs once API service discovery every 30s, around 120 requests per minute.
Verified this bug.
console image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f2ceacf044e29e75c488e5e641598015b2fff97274ab898409be5e020fea029
io.openshift.build.commit.id=d86dbff35d53acd8a0a48cde98cb6037d9577edb