Created attachment 1249446[details]
partial log of oc get secrets with loglevel=6
Description of problem:
In OCP 3.5 in a large cluster (300 nodes, 1000 projects, 5000 pods, 7000 builds, 30000 secrets, etc), oc get commands on resources such as these takes many minutes. The commands do eventually complete successfully.
5000 pods: 6m 26s
7000 builds: 5m 31s
30000 secrets: 44m 39s
In 3.4 these commands took seconds
Note: strangely, oc get for these resources with -w to set a watch returns the full list very quickly - on the order of what I expect.
Running oc get with --loglevel 6 shows a repeated pattern of errors with too many files open/failed to write to cache (see below in additional info)
Version: 3.5.0.17
How reproducible: Always in a cluster with large #s of resources
Steps to Reproduce:
1. Create a cluster with 1000 projects and create 30 secrets in each project
2. oc get secrets --all-namespaces
Actual results:
Command will take 40+ minutes to run
Expected results:
Command runs in a few seconds
Additional info:
Running with loglevel=6 shows a repeated throttling/failed to write to cache/too many files open pattern. See the attached log
Created attachment 1249446 [details] partial log of oc get secrets with loglevel=6 Description of problem: In OCP 3.5 in a large cluster (300 nodes, 1000 projects, 5000 pods, 7000 builds, 30000 secrets, etc), oc get commands on resources such as these takes many minutes. The commands do eventually complete successfully. 5000 pods: 6m 26s 7000 builds: 5m 31s 30000 secrets: 44m 39s In 3.4 these commands took seconds Note: strangely, oc get for these resources with -w to set a watch returns the full list very quickly - on the order of what I expect. Running oc get with --loglevel 6 shows a repeated pattern of errors with too many files open/failed to write to cache (see below in additional info) Version: 3.5.0.17 How reproducible: Always in a cluster with large #s of resources Steps to Reproduce: 1. Create a cluster with 1000 projects and create 30 secrets in each project 2. oc get secrets --all-namespaces Actual results: Command will take 40+ minutes to run Expected results: Command runs in a few seconds Additional info: Running with loglevel=6 shows a repeated throttling/failed to write to cache/too many files open pattern. See the attached log