Description of problem: After a project is created in OpenShift, the /apis/project.openshift.io/v1/projects endpoint does not display the newly created project. I've noticed this happening only in the starter online clusters (it doesn't happen with Minishift for example). Version-Release number of selected component (if applicable): OpenShift Master: v3.10.0-0.69.0 Kubernetes Master: v1.10.0+b81c8f8 OpenShift Web Console: v3.10.0-0.69.0 How reproducible: always Steps to Reproduce: 1. Make sure you have no projects in your starter cluster (eg. https://console.starter-us-east-1.openshift.com) 2. Open a terminal and type: watch "http https://console.starter-us-east-1.openshift.com/apis/project.openshift.io/v1/projects Authorization:Bearer\ YOUR TOKEN" You should see the following result: {"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[]} 3. Create a project in the Web Console (or using oc new-project) oc new-project foo Actual results: {"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[]} Expected results: {"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[{"metadata":{"name":"foo","selfLink":"/apis/project.ope nshift.io/v1/projects/foo","uid":"cb3e9bd3-78aa-11e8-9301-1250f17a13c8","resourceVersion":"2598305553","creationTimestamp":"2018-06-25T19:05:58Z","labels":{"openshift.io/hibernate-include":" true"},"annotations":{"openshift.io/description":"","openshift.io/display-name":"","openshift.io/requester":"rhn-support-ggastald","openshift.io/sa.scc.mcs":"s0:c108,c2","openshift.io/sa.scc .supplemental-groups":"1011560000/10000","openshift.io/sa.scc.uid-range":"1011560000/10000"}},"spec":{"finalizers":["openshift.io/origin","kubernetes"]},"status":{"phase":"Active"}}]} Additional Info: I noticed this same behavior in the web console too (the project disappears and if I try to create a project again I get an error saying that my quota is not enough)
I noticed that in general it takes ~ 1 minute to return the correct data in the /projects endpoint
6132 projects on starter-us-east-1 (openshift v3.11.0-0.21.0). What I see: $ time oc delete project starxia-proj --loglevel 6 I0913 11:17:01.961725 13833 loader.go:359] Config loaded from file /home/xxia/.kube/config I0913 11:17:01.962930 13833 loader.go:359] Config loaded from file /home/xxia/.kube/config I0913 11:17:01.966040 13833 loader.go:359] Config loaded from file /home/xxia/.kube/config I0913 11:17:01.975222 13833 loader.go:359] Config loaded from file /home/xxia/.kube/config I0913 11:17:01.977166 13833 loader.go:359] Config loaded from file /home/xxia/.kube/config I0913 11:17:01.979135 13833 loader.go:359] Config loaded from file /home/xxia/.kube/config I0913 11:17:03.189714 13833 round_trippers.go:405] DELETE https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects/starxia-proj 200 OK in 1210 milliseconds project.project.openshift.io "starxia-proj" deleted I0913 11:17:03.190231 13833 delete.go:272] object does not implement the Object interfaces I0913 11:17:03.450450 13833 round_trippers.go:405] GET https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects/starxia-proj 200 OK in 259 milliseconds I0913 11:17:03.719768 13833 round_trippers.go:405] GET https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects?fieldSelector=metadata.name%3Dstarxia-proj&resourceVersion=3135626418&watch=true 200 OK in 268 milliseconds real 2m55.017s user 0m0.212s sys 0m0.020s
The issue is not happened on INT(free-int) and Prod Pro(pro-us-east-1) and the correct data can be got in 6s. From the description, it is more like a perfermance issue. @Mike, could you help take a look whether the DB searching time is effected by tons of projects?
Moving this to 3.11.x in case we still see this behavior.
No new reports/occurrences against 3.11 once we upgraded there and the 3.x clusters have been retired. Closing for now in case new reports occur.