Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1594939

Summary: Projects endpoint in a Starter cluster returns empty list of projects after a project is created
Product: OpenShift Container Platform Reporter: George Gastaldi <ggastald>
Component: openshift-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Mike Fiedler <mifiedle>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.11.0CC: aos-bugs, ggastald, jforrest, jokerman, mfojtik, mifiedle, mmccomas, scuppett, xxia, yufchang
Target Milestone: ---Keywords: OnlineStarter
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-12 11:16:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description George Gastaldi 2018-06-25 19:10:16 UTC
Description of problem:

After a project is created in OpenShift, the /apis/project.openshift.io/v1/projects endpoint does not display the newly created project. I've noticed this happening only in the starter online clusters (it doesn't happen with Minishift for example).


Version-Release number of selected component (if applicable):

OpenShift Master: v3.10.0-0.69.0
Kubernetes Master: v1.10.0+b81c8f8
OpenShift Web Console: v3.10.0-0.69.0


How reproducible: always


Steps to Reproduce:

1. Make sure you have no projects in your starter cluster (eg. https://console.starter-us-east-1.openshift.com)

2. Open a terminal and type:

    watch "http https://console.starter-us-east-1.openshift.com/apis/project.openshift.io/v1/projects Authorization:Bearer\ YOUR TOKEN"

You should see the following result:

{"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[]}

3. Create a project in the Web Console (or using oc new-project) 

     oc new-project foo

Actual results:
{"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[]}


Expected results:

{"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[{"metadata":{"name":"foo","selfLink":"/apis/project.ope
nshift.io/v1/projects/foo","uid":"cb3e9bd3-78aa-11e8-9301-1250f17a13c8","resourceVersion":"2598305553","creationTimestamp":"2018-06-25T19:05:58Z","labels":{"openshift.io/hibernate-include":"
true"},"annotations":{"openshift.io/description":"","openshift.io/display-name":"","openshift.io/requester":"rhn-support-ggastald","openshift.io/sa.scc.mcs":"s0:c108,c2","openshift.io/sa.scc
.supplemental-groups":"1011560000/10000","openshift.io/sa.scc.uid-range":"1011560000/10000"}},"spec":{"finalizers":["openshift.io/origin","kubernetes"]},"status":{"phase":"Active"}}]}


Additional Info: 

I noticed this same behavior in the web console too (the project disappears and if I try to create a project again I get an error saying that my quota is not enough)

Comment 1 George Gastaldi 2018-06-25 19:12:32 UTC
I noticed that in general it takes ~ 1 minute to return the correct data in the /projects endpoint

Comment 2 Xingxing Xia 2018-09-13 03:23:28 UTC
6132 projects on starter-us-east-1 (openshift v3.11.0-0.21.0). What I see:
$ time oc delete project starxia-proj --loglevel 6
I0913 11:17:01.961725   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.962930   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.966040   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.975222   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.977166   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.979135   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:03.189714   13833 round_trippers.go:405] DELETE https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects/starxia-proj 200 OK in 1210 milliseconds
project.project.openshift.io "starxia-proj" deleted
I0913 11:17:03.190231   13833 delete.go:272] object does not implement the Object interfaces
I0913 11:17:03.450450   13833 round_trippers.go:405] GET https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects/starxia-proj 200 OK in 259 milliseconds
I0913 11:17:03.719768   13833 round_trippers.go:405] GET https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects?fieldSelector=metadata.name%3Dstarxia-proj&resourceVersion=3135626418&watch=true 200 OK in 268 milliseconds

real	2m55.017s
user	0m0.212s
sys	0m0.020s

Comment 3 yasun 2018-09-13 11:35:53 UTC
The issue is not happened on INT(free-int) and Prod Pro(pro-us-east-1) and the correct data can be got in 6s. From the description, it is more like a perfermance issue. @Mike, could you help take a look whether the DB searching time is effected by tons of projects?

Comment 4 Michal Fojtik 2019-04-01 09:10:36 UTC
Moving this to 3.11.x in case we still see this behavior.

Comment 5 Stephen Cuppett 2019-08-12 11:16:46 UTC
No new reports/occurrences against 3.11 once we upgraded there and the 3.x clusters have been retired. Closing for now in case new reports occur.