Bug 1594939 - Projects endpoint in a Starter cluster returns empty list of projects after a project is created
Summary: Projects endpoint in a Starter cluster returns empty list of projects after a...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-apiserver
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.11.z
Assignee: Stefan Schimanski
QA Contact: Mike Fiedler
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-25 19:10 UTC by George Gastaldi
Modified: 2019-08-12 11:16 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-12 11:16:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description George Gastaldi 2018-06-25 19:10:16 UTC
Description of problem:

After a project is created in OpenShift, the /apis/project.openshift.io/v1/projects endpoint does not display the newly created project. I've noticed this happening only in the starter online clusters (it doesn't happen with Minishift for example).


Version-Release number of selected component (if applicable):

OpenShift Master: v3.10.0-0.69.0
Kubernetes Master: v1.10.0+b81c8f8
OpenShift Web Console: v3.10.0-0.69.0


How reproducible: always


Steps to Reproduce:

1. Make sure you have no projects in your starter cluster (eg. https://console.starter-us-east-1.openshift.com)

2. Open a terminal and type:

    watch "http https://console.starter-us-east-1.openshift.com/apis/project.openshift.io/v1/projects Authorization:Bearer\ YOUR TOKEN"

You should see the following result:

{"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[]}

3. Create a project in the Web Console (or using oc new-project) 

     oc new-project foo

Actual results:
{"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[]}


Expected results:

{"kind":"ProjectList","apiVersion":"project.openshift.io/v1","metadata":{"selfLink":"/apis/project.openshift.io/v1/projects"},"items":[{"metadata":{"name":"foo","selfLink":"/apis/project.ope
nshift.io/v1/projects/foo","uid":"cb3e9bd3-78aa-11e8-9301-1250f17a13c8","resourceVersion":"2598305553","creationTimestamp":"2018-06-25T19:05:58Z","labels":{"openshift.io/hibernate-include":"
true"},"annotations":{"openshift.io/description":"","openshift.io/display-name":"","openshift.io/requester":"rhn-support-ggastald","openshift.io/sa.scc.mcs":"s0:c108,c2","openshift.io/sa.scc
.supplemental-groups":"1011560000/10000","openshift.io/sa.scc.uid-range":"1011560000/10000"}},"spec":{"finalizers":["openshift.io/origin","kubernetes"]},"status":{"phase":"Active"}}]}


Additional Info: 

I noticed this same behavior in the web console too (the project disappears and if I try to create a project again I get an error saying that my quota is not enough)

Comment 1 George Gastaldi 2018-06-25 19:12:32 UTC
I noticed that in general it takes ~ 1 minute to return the correct data in the /projects endpoint

Comment 2 Xingxing Xia 2018-09-13 03:23:28 UTC
6132 projects on starter-us-east-1 (openshift v3.11.0-0.21.0). What I see:
$ time oc delete project starxia-proj --loglevel 6
I0913 11:17:01.961725   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.962930   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.966040   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.975222   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.977166   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:01.979135   13833 loader.go:359] Config loaded from file /home/xxia/.kube/config
I0913 11:17:03.189714   13833 round_trippers.go:405] DELETE https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects/starxia-proj 200 OK in 1210 milliseconds
project.project.openshift.io "starxia-proj" deleted
I0913 11:17:03.190231   13833 delete.go:272] object does not implement the Object interfaces
I0913 11:17:03.450450   13833 round_trippers.go:405] GET https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects/starxia-proj 200 OK in 259 milliseconds
I0913 11:17:03.719768   13833 round_trippers.go:405] GET https://api.starter-us-east-1.openshift.com:443/apis/project.openshift.io/v1/projects?fieldSelector=metadata.name%3Dstarxia-proj&resourceVersion=3135626418&watch=true 200 OK in 268 milliseconds

real	2m55.017s
user	0m0.212s
sys	0m0.020s

Comment 3 yasun 2018-09-13 11:35:53 UTC
The issue is not happened on INT(free-int) and Prod Pro(pro-us-east-1) and the correct data can be got in 6s. From the description, it is more like a perfermance issue. @Mike, could you help take a look whether the DB searching time is effected by tons of projects?

Comment 4 Michal Fojtik 2019-04-01 09:10:36 UTC
Moving this to 3.11.x in case we still see this behavior.

Comment 5 Stephen Cuppett 2019-08-12 11:16:46 UTC
No new reports/occurrences against 3.11 once we upgraded there and the 3.x clusters have been retired. Closing for now in case new reports occur.


Note You need to log in before you can comment on or make changes to this bug.