Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1648110

Summary: user's project list is empty sometimes
Product: OpenShift Container Platform Reporter: Alex Chvatal <achvatal>
Component: openshift-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED DEFERRED QA Contact: Xingxing Xia <xxia>
Severity: high Docs Contact:
Priority: medium    
Version: 3.9.0CC: achvatal, aos-bugs, bmchugh, jokerman, mfojtik, mirollin, mmccomas, scuppett
Target Milestone: ---Keywords: OpsBlocker
Target Release: 3.9.z   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-20 18:56:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Chvatal 2018-11-08 20:54:06 UTC 
Description of problem:
When loading up the web console, the user's projects aren't displayed until they reload the page.

The user sees the same behavior when using `oc get projects` - sometimes the project list is empty and sometimes it lists projects.

When the customer tries to delete the user from the membership of the projects, the browser shows around 8126 requests, making their browser hang.


Version-Release number of selected component (if applicable):
v3.9.41

How reproducible:
It seems unpredictable but frequent, around 50% of the calls to `oc get projects`

Steps to Reproduce:
1. Run `oc get projects` or load the web console

Actual results:
No projects listed

Expected results:
Projects listed
Comment 8 Alex Chvatal 2018-11-08 21:03:50 UTC 
Created attachment 1503467 [details]
multiple `oc projects` output pt 2
Comment 15 Alex Chvatal 2018-11-12 14:54:31 UTC 
We've moved the masters' /var/lib/etcd directories onto their own volumes. It looks like that's alleviating this issue, and we're going to confirm with the customer.
Comment 18 Alex Chvatal 2018-11-13 19:50:39 UTC 
Scratch my last, the issue has not been resolved by moving etcd onto its own volume.

It sounds like the customer has a cron-db-backup pod that is created every hour, which is using a _lot_ of network traffic. 3 mins past every hour, the cluster is basically unusable for 10/11 mins.  etcd, api - they are saturating their Direct connect link.
Comment 20 Stephen Cuppett 2019-11-20 18:56:10 UTC 
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift