We are hitting this issue on 4.6.8 with ~ 160 CRDs
Sorry, I had browser issues when uploading my first comment. We did a oc get crd -A and got 160 CustomResourceDefinitions but not that much CustomResources. Here is the output of the python script: python2 list_all.py -c -s all -o count hostsubnets.network.openshift.io: 6 operators.operators.coreos.com: 6 kibanas.logging.openshift.io: 0 securitycontextconstraints.security.openshift.io: 9 servicemeshcontrolplanes.maistra.io: 0 clusterserviceversions.operators.coreos.com: 2 elasticsearches.logging.openshift.io: 0 consoleclidownloads.console.openshift.io: 3 clusterversions.config.openshift.io: 1 dnsrecords.ingress.operator.openshift.io: 0 tridentversions.trident.netapp.io: 0 machinehealthchecks.machine.openshift.io: 0 clusterautoscalers.autoscaling.openshift.io: 0 configs.imageregistry.operator.openshift.io: 1 knativeeventings.operator.knative.dev: 0 profiles.tuned.openshift.io: 0 etcds.operator.openshift.io: 1 tridenttransactions.trident.netapp.io: 0 servicecas.operator.openshift.io: 1 kubeletconfigs.machineconfiguration.openshift.io: 0 storageversionmigrations.migration.k8s.io: 0 storages.operator.openshift.io: 1 catalogsources.operators.coreos.com: 0 consoleyamlsamples.console.openshift.io: 0 clusteroperators.config.openshift.io: 30 configs.samples.operator.openshift.io: 1 servicemeshmemberrolls.maistra.io: 0 volumesnapshots.snapshot.storage.k8s.io: 0 credentialsrequests.cloudcredential.openshift.io: 0 ingresses.networking.internal.knative.dev: 0 tridentvolumes.trident.netapp.io: 0 imagepruners.imageregistry.operator.openshift.io: 1 services.serving.knative.dev: 0 operatorgroups.operators.coreos.com: 0 dnses.config.openshift.io: 1 consoles.operator.openshift.io: 1 authentications.operator.openshift.io: 1 provisionings.metal3.io: 0 configs.operator.openshift.io: 1 apiservers.config.openshift.io: 1 openshiftapiservers.operator.openshift.io: 1 projects.config.openshift.io: 1 revisions.serving.knative.dev: 0 networks.config.openshift.io: 1 cloudcredentials.operator.openshift.io: 1 clusterlogforwarders.logging.openshift.io: 0 egressnetworkpolicies.network.openshift.io: 0 featuregates.config.openshift.io: 1 imagecontentsourcepolicies.operator.openshift.io: 0 alertmanagers.monitoring.coreos.com: 0 consoleexternalloglinks.console.openshift.io: 1 authentications.config.openshift.io: 1 ippools.whereabouts.cni.cncf.io: 0 machineconfigs.machineconfiguration.openshift.io: 20 rangeallocations.security.internal.openshift.io: 1 volumesnapshotclasses.snapshot.storage.k8s.io: 1 overlappingrangeipreservations.whereabouts.cni.cncf.io: 0 serverlessservices.networking.internal.knative.dev: 0 tuneds.tuned.openshift.io: 0 servicemonitors.monitoring.coreos.com: 0 machines.machine.openshift.io: 0 volumesnapshotcontents.snapshot.storage.k8s.io: 0 metrics.autoscaling.internal.knative.dev: 0 knativeservings.operator.knative.dev: 0 tridentsnapshots.trident.netapp.io: 0 operatorhubs.config.openshift.io: 1 kubeschedulers.operator.openshift.io: 1 thanosrulers.monitoring.coreos.com: 0 consolelinks.console.openshift.io: 3 netnamespaces.network.openshift.io: 68 subscriptions.operators.coreos.com: 0 csisnapshotcontrollers.operator.openshift.io: 1 machineautoscalers.autoscaling.openshift.io: 0 openshiftcontrollermanagers.operator.openshift.io: 1 images.caching.internal.knative.dev: 0 clusterresourcequotas.quota.openshift.io: 0 installplans.operators.coreos.com: 0 images.config.openshift.io: 1 controllerconfigs.machineconfiguration.openshift.io: 1 clusterloggings.logging.openshift.io: 0 configurations.serving.knative.dev: 0 servicemeshmembers.maistra.io: 0 builds.config.openshift.io: 1 tridentnodes.trident.netapp.io: 0 proxies.config.openshift.io: 1 consoles.config.openshift.io: 1 helmchartrepositories.helm.openshift.io: 1 clustercsidrivers.operator.openshift.io: 0 probes.monitoring.coreos.com: 0 routes.serving.knative.dev: 0 tridentbackends.trident.netapp.io: 0 schedulers.config.openshift.io: 1 prometheuses.monitoring.coreos.com: 0 ingresses.config.openshift.io: 1 kubecontrollermanagers.operator.openshift.io: 1 tridentprovisioners.trident.netapp.io: 0 clusternetworks.network.openshift.io: 1 kubeapiservers.operator.openshift.io: 1 operatorpkis.network.operator.openshift.io: 0 dnses.operator.openshift.io: 1 oauths.config.openshift.io: 1 machineconfigpools.machineconfiguration.openshift.io: 2 tridentstorageclasses.trident.netapp.io: 0 storagestates.migration.k8s.io: 0 kubestorageversionmigrators.operator.openshift.io: 1 baremetalhosts.metal3.io: 0 infrastructures.config.openshift.io: 1 networks.operator.openshift.io: 1 prometheusrules.monitoring.coreos.com: 0 ingresscontrollers.operator.openshift.io: 0 rolebindingrestrictions.authorization.openshift.io: 0 certificates.networking.internal.knative.dev: 0 machinesets.machine.openshift.io: 0 containerruntimeconfigs.machineconfiguration.openshift.io: 0 podautoscalers.autoscaling.internal.knative.dev: 0 network-attachment-definitions.k8s.cni.cncf.io: 0 knativekafkas.operator.serverless.openshift.io: 0 podmonitors.monitoring.coreos.com: 0 consolenotifications.console.openshift.io: 0 We are always hitting the throttling and have oc hanging for ~10 seconds. We have a support ticket open which led me here :)
*** Bug 1918675 has been marked as a duplicate of this bug. ***
Hello, Could you please give any estimation on the date the fix will be available for 4.6.z? Thanks in advance.
(In reply to mchebbi from comment #13) > Hello, > > Could you please give any estimation on the date the fix will be available > for 4.6.z? > Thanks in advance. I'm hoping to put together a PR today so it will most likely be one of the next .z streams.
(In reply to Maciej Szulik from comment #14) > (In reply to mchebbi from comment #13) > > Hello, > > > > Could you please give any estimation on the date the fix will be available > > for 4.6.z? > > Thanks in advance. > > I'm hoping to put together a PR today so it will most likely be one of the > next .z streams. Thanks for your reply.
Verified on 4.6.16 My OOTB cluster (no optional operators) has 93 CRD. When I add 175 additional custom CRD with 5 resources each (268 CRD) I do not see throttling messages. When I add more than 175 additional CRD I start to see throttling messages on get/delete especially.
[xxx@xxx ~]$ oc get operators I0204 12:52:11.814344 2467516 request.go:645] Throttling request took 1.17338979s, request: GET:https://api.xxxx.xxxxx.xxxxx:6443/apis/autoscaling.openshift.io/v1?timeout=32s NAME AGE cluster-logging.openshift-logging 51d elasticsearch-operator.openshift-operators-redhat 51d grafana-operator.tkstede 8d jaeger-product.openshift-operators 51d kiali.openshift-operators 51d openshiftartifactoryha-operator.p224075-test23 36d percona-server-mongodb-operator-certified.intranet-mongodb 43d serverless-operator.openshift-serverless 51d servicemeshoperator.openshift-operators 51d [xxx@xxx ~]$ oc get crd | wc -l 168 This happens on one of our rather "fresh" clusters which is on 4.6.13
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.6.16 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0308
Reopening this bug. It appears customers are still hitting this bug in 4.7.24: $ oc adm must-gather [must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:040f48c020420ff93b 227216469f6c2971cf10fac2b0b52ea9853e88ec1964a6 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested informat ion. ClusterID: 670fc965-c512-44dd-b005-d8e418008e33 ClusterVersion: Stable at "4.7.24" ClusterOperators: All healthy and stable [must-gather ] OUT namespace/openshift-must-gather-mwvtr created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-wwxrm created [must-gather ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:040f48c020420ff93b227216469f6 c2971cf10fac2b0b52ea9853e88ec1964a6 created [must-gather-h6fd7] POD 2021-11-29T18:52:10.344745676Z I1129 18:52:10.344469 52 request.go:655] Throttling request took 1.18 9284656s, request: GET:https://198.223.0.1:443/apis/hostpathprovisioner.kubevirt.io/v1alpha1?timeout=32s [must-gather-h6fd7] POD 2021-11-29T18:52:14.564983722Z Gathering data for ns/openshift-cluster-version... [must-gather-h6fd7] POD 2021-11-29T18:52:20.389993859Z I1129 18:52:20.389874 52 request.go:655] Throttling request took 4.79 4315983s, request: GET:https://198.223.0.1:443/apis/machine.openshift.io/v1beta1?timeout=32s [must-gather-h6fd7] POD 2021-11-29T18:52:21.236575249Z Gathering data for ns/default... [must-gather-h6fd7] POD 2021-11-29T18:52:26.904035255Z Gathering data for ns/openshift... [must-gather-h6fd7] POD 2021-11-29T18:52:30.535553160Z I1129 18:52:30.535507 52 request.go:655] Throttling request took 3.59 5756696s, request: GET:https://198.223.0.1:443/apis/cdi.kubevirt.io/v1beta1?timeout=32s [must-gather-h6fd7] POD 2021-11-29T18:52:32.603482444Z Gathering data for ns/kube-system... [must-gather-h6fd7] POD 2021-11-29T18:52:51.331727171Z I1129 18:52:51.331690 52 request.go:655] Throttling request took 1.19 5777303s, request: GET:https://198.223.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s [must-gather-h6fd7] POD 2021-11-29T18:52:56.133105402Z Gathering data for ns/openshift-etcd... [must-gather-h6fd7] POD 2021-11-29T18:53:03.028319229Z I1129 18:53:03.028274 52 request.go:655] Throttling request took 1.19 3998933s, request: GET:https://198.223.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s [must-gather-h6fd7] POD 2021-11-29T18:53:07.715532155Z Gathering data for ns/openshift-kni-infra... [must-gather-h6fd7] POD 2021-11-29T18:54:55.419257191Z I1129 18:54:55.419206 52 request.go:655] Throttling request took 1.19 5567744s, request: GET:https://198.223.0.1:443/apis/ingress.operator.openshift.io/v1?timeout=32s This is one of their smaller clusters with 172 crds: $ omg get crds | wc -l 172 @maszulik
(In reply to Dan Small from comment #28) > This is one of their smaller clusters with 172 crds: > $ omg get crds | wc -l > 172 Discovery is built-in resources and CRDs. You can easily verify the total number by running: $ oc api-resources |wc -l 237 $ oc get crds|wc -l 135 Those are data from my 4.9 cluster, as you see with barely 135 CRDs, I'm at 250 limit. We are bumping that limit even higher in 4.10, but I'm not envisioning backporting that.
This should be tracked by a new bug, not reopening a bug already included in a shipped errata. The current code change is tested and shipped. Moving this back to closed->errata. I will clone this bug to track any additional code changes. cc: @dansmall @sgordon
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days