Created attachment 1887205 [details] Reproducer resource Created attachment 1887205 [details] Reproducer resource Description of problem: Listing secrets in all namespaces with a specific labelSelector does not work properly: rsevilla@wonderland /tmp $ oc get secret -A -l kube-burner-job=cluster-density -v9 I0606 16:37:17.875754 2681352 loader.go:372] Config loaded from file: /home/rsevilla/kubeconfig I0606 16:37:17.884142 2681352 round_trippers.go:466] curl -v -XGET -H "User-Agent: oc/4.10.0 (linux/amd64) kubernetes/04ad1b5" -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=kube-burner-job%3Dcluster-density&limit=500' I0606 16:37:17.885481 2681352 round_trippers.go:495] HTTP Trace: DNS Lookup for api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com resolved to [{34.208.8.2 } {44.224.254.41 } {44.242.34.69 } {54.218.246.157 }] I0606 16:37:18.094176 2681352 round_trippers.go:510] HTTP Trace: Dial to tcp:34.208.8.2:6443 succeed I0606 16:37:18.580598 2681352 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 208 ms TLSHandshake 214 ms ServerProcessing 271 ms Duration 696 ms I0606 16:37:18.580656 2681352 round_trippers.go:577] Response Headers: I0606 16:37:18.580682 2681352 round_trippers.go:580] Cache-Control: no-cache, private I0606 16:37:18.580703 2681352 round_trippers.go:580] Content-Type: application/json I0606 16:37:18.580722 2681352 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 7eb65202-ec98-465b-b7d4-eea85843f875 I0606 16:37:18.580742 2681352 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 64cc9fb8-ee32-4d93-bb3f-f10697c70027 I0606 16:37:18.580761 2681352 round_trippers.go:580] Content-Length: 1610 I0606 16:37:18.580779 2681352 round_trippers.go:580] Date: Mon, 06 Jun 2022 14:37:18 GMT I0606 16:37:18.580798 2681352 round_trippers.go:580] Audit-Id: ec6343eb-5924-4f28-aeb7-2d67fa181467 I0606 16:37:18.580893 2681352 request.go:1181] Response Body: {"kind":"Table","apiVersion":"meta.k8s.io/v1","metadata":{"resourceVersion":"1154448"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names","priority":0},{"name":"Type","type":"string","format":"","description":"Used to facilitate programmatic handling of secret data. More info: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types","priority":0},{"name":"Data","type":"string","format":"","description":"Data contains the secret data. Each key must consist of alphanumeric characters, '-', '_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in https://tools.ietf.org/html/rfc4648#section-4","priority":0},{"name":"Age","type":"string","format":"","description":"CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.\n\nPopulated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata","priority":0}],"rows":[]} No resources found rsevilla@wonderland /tmp $ ./oc get secret --no-headers -l kube-burner-job=cluster-density -n cluster-density-1 --show-labels cluster-density-1 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-10 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-2 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-3 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-4 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-5 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-6 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-7 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-8 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-9 Opaque 1 45m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 In the cluster above there were 100 secrets created across 100 namespaces, all of them sharing the label kube-burner-job=cluster-density On the other hand, listing configmaps, (which have similar labels worked fine) as shown below rsevilla@wonderland /tmp $ ./oc get cm --no-headers -A -l kube-burner-job=cluster-density | wc -l 100 Version-Release number of selected component (if applicable): rsevilla@wonderland /tmp $ ./oc version Client Version: 4.11.0-0.nightly-2022-06-04-014713 Kustomize Version: v4.5.4 Server Version: 4.11.0-0.nightly-2022-06-04-014713 Kubernetes Version: v1.24.0+bb9c2f1 Steps to Reproduce: 1. $ for N in {1..10}; do export N ; envsubst < ns.yml | oc apply -f - & done (ns.yml is attached to this BZ) 2. $ oc get secret -A -l foo=bar | wc -l 0 Expected results: Oc lists all objects Additional info: After deleting 9 namespaces out of 10, the listing command works rsevilla@wonderland /tmp $ ./oc delete ns cluster-density-{2..10} namespace "cluster-density-2" deleted namespace "cluster-density-3" deleted namespace "cluster-density-4" deleted namespace "cluster-density-5" deleted namespace "cluster-density-6" deleted namespace "cluster-density-7" deleted namespace "cluster-density-8" deleted namespace "cluster-density-9" deleted namespace "cluster-density-10" deleted rsevilla@wonderland /tmp $ ./oc get secret --no-headers -l kube-burner-job=cluster-density -n cluster-density-1 --show-labels cluster-density-1 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-10 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-2 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-3 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-4 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-5 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-6 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-7 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-8 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4 cluster-density-9 Opaque 1 50m kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
I can't reproduce the problem in vanilla kubernetes, it seems the limit variable in openshift behaves differently. A more generic reproducder, 1. Create 100 fake secrets $ cat secret.yaml apiVersion: v1 kind: Secret metadata: generateName: mysecret- labels: test: test type: Opaque data: $ oc create namespace testns $ for i in `seq 1 100` ; do oc create -f secret.yaml -n testns; done 2. Try to list the secrets in all namespaces filtering by the label used, with the default limit variable = 500 oc get --raw 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=test%3Dtest&limit=500' | jq .items[].metadata.name | wc 0 0 0 ---> No resources 3. List the secrets without limit or with a higher limit value oc get --raw 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=test%3Dtest&limit=687' | jq .items[].metadata.name | wc 100 100 1700 oc get --raw 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=test%3Dtest' | jq .items[].metadata.name | wc 100 100 1700
Created attachment 1893215 [details] Not kube-burner reproducer
(In reply to Antonio Ojea from comment #5) > To be clear, I'm not able to reproduce it in a cluster where kube-burner > didn't run > > Next step is to reproduce it without kube-burner > > 1. Create 100 fake secrets > > $ cat secret.yaml > apiVersion: v1 > kind: Secret > metadata: > generateName: mysecret- > labels: > test: test > type: Opaque > data: > > $ oc create namespace testns > > $ for i in `seq 1 100` ; do oc create -f secret.yaml -n testns; done > > 2. Try to list the secrets in all namespaces filtering by the label used, > with the default limit variable = 500 > > oc get secrets -A | wc -l > > > > if this is not reproducible we have to understand what are the differences ( Hey!, I did reproduce this issue w/o kube-burner using the reproducer I just attached (reproducer.sh), that creates 50 namespaces with 10 secrets labeled with foo=bar each: rsevilla@wonderland /tmp $ ./reproducer.sh > /dev/null rsevilla@wonderland /tmp $ oc get secret -A -l foo=bar | wc -l 79 rsevilla@wonderland /tmp $ kubectl get secret -A -l foo=bar | wc -l 79 rsevilla@wonderland /tmp $ oc version kubeClient Version: 4.10.15 Server Version: 4.11.0-0.nightly-2022-06-15-222801 Kubernetes Version: v1.24.0+25f9057 rsevilla@wonderland /tmp $ kubectl version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.24.0 Kustomize Version: v4.5.4 Server Version: v1.24.0+25f9057
Seeing the same thing with a newer client and server versions % oc version Client Version: 4.11.0-0.nightly-2022-06-21-094850 Kustomize Version: v4.5.4 Server Version: 4.11.0-0.nightly-2022-06-25-132614 Kubernetes Version: v1.24.0+9ddc8b1 % kubectl version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.24.1 Kustomize Version: v4.5.4 Server Version: v1.24.0+9ddc8b1 % oc get secrets -A -l foo=bar NAMESPACE NAME TYPE DATA AGE foo-37 fubar-8 Opaque 0 3m41s foo-37 fubar-9 Opaque 0 3m40s foo-38 fubar-1 Opaque 0 3m41s foo-38 fubar-10 Opaque 0 3m40s foo-38 fubar-2 Opaque 0 3m41s foo-38 fubar-3 Opaque 0 3m41s foo-38 fubar-4 Opaque 0 3m41s foo-38 fubar-5 Opaque 0 3m41s foo-38 fubar-6 Opaque 0 3m41s foo-38 fubar-7 Opaque 0 3m41s foo-38 fubar-8 Opaque 0 3m40s foo-38 fubar-9 Opaque 0 3m40s foo-39 fubar-1 Opaque 0 3m41s foo-39 fubar-10 Opaque 0 3m40s foo-39 fubar-2 Opaque 0 3m41s foo-39 fubar-3 Opaque 0 3m41s foo-39 fubar-4 Opaque 0 3m41s foo-39 fubar-5 Opaque 0 3m41s foo-39 fubar-6 Opaque 0 3m41s foo-39 fubar-7 Opaque 0 3m41s foo-39 fubar-8 Opaque 0 3m40s foo-39 fubar-9 Opaque 0 3m40s foo-4 fubar-1 Opaque 0 3m49s foo-4 fubar-10 Opaque 0 3m48s foo-4 fubar-2 Opaque 0 3m49s foo-4 fubar-3 Opaque 0 3m49s foo-4 fubar-4 Opaque 0 3m49s foo-4 fubar-5 Opaque 0 3m49s foo-4 fubar-6 Opaque 0 3m49s foo-4 fubar-7 Opaque 0 3m49s foo-4 fubar-8 Opaque 0 3m49s foo-4 fubar-9 Opaque 0 3m48s foo-40 fubar-1 Opaque 0 3m41s foo-40 fubar-10 Opaque 0 3m40s foo-40 fubar-2 Opaque 0 3m41s foo-40 fubar-3 Opaque 0 3m41s foo-40 fubar-4 Opaque 0 3m41s foo-40 fubar-5 Opaque 0 3m40s foo-40 fubar-6 Opaque 0 3m40s foo-40 fubar-7 Opaque 0 3m40s foo-40 fubar-8 Opaque 0 3m40s foo-40 fubar-9 Opaque 0 3m40s foo-41 fubar-1 Opaque 0 3m39s foo-41 fubar-10 Opaque 0 3m39s foo-41 fubar-2 Opaque 0 3m39s foo-41 fubar-3 Opaque 0 3m39s foo-41 fubar-4 Opaque 0 3m39s foo-41 fubar-5 Opaque 0 3m39s foo-41 fubar-6 Opaque 0 3m39s foo-41 fubar-7 Opaque 0 3m39s foo-41 fubar-8 Opaque 0 3m39s foo-41 fubar-9 Opaque 0 3m39s foo-42 fubar-1 Opaque 0 3m39s foo-42 fubar-10 Opaque 0 3m38s foo-42 fubar-2 Opaque 0 3m39s foo-42 fubar-3 Opaque 0 3m39s foo-42 fubar-4 Opaque 0 3m39s foo-42 fubar-5 Opaque 0 3m39s foo-42 fubar-6 Opaque 0 3m39s foo-42 fubar-7 Opaque 0 3m38s foo-42 fubar-8 Opaque 0 3m38s foo-42 fubar-9 Opaque 0 3m38s foo-43 fubar-1 Opaque 0 3m40s foo-43 fubar-10 Opaque 0 3m39s foo-43 fubar-2 Opaque 0 3m39s foo-43 fubar-3 Opaque 0 3m39s foo-43 fubar-4 Opaque 0 3m39s foo-43 fubar-5 Opaque 0 3m39s foo-43 fubar-6 Opaque 0 3m39s foo-43 fubar-7 Opaque 0 3m39s foo-43 fubar-8 Opaque 0 3m39s foo-43 fubar-9 Opaque 0 3m39s foo-44 fubar-1 Opaque 0 3m39s foo-44 fubar-10 Opaque 0 3m38s foo-44 fubar-2 Opaque 0 3m39s foo-44 fubar-3 Opaque 0 3m38s foo-44 fubar-4 Opaque 0 3m38s foo-44 fubar-5 Opaque 0 3m38s foo-44 fubar-6 Opaque 0 3m38s foo-44 fubar-7 Opaque 0 3m38s foo-44 fubar-8 Opaque 0 3m38s foo-44 fubar-9 Opaque 0 3m38s foo-45 fubar-1 Opaque 0 3m39s foo-45 fubar-10 Opaque 0 3m38s foo-45 fubar-2 Opaque 0 3m39s foo-45 fubar-3 Opaque 0 3m39s foo-45 fubar-4 Opaque 0 3m39s foo-45 fubar-5 Opaque 0 3m38s foo-45 fubar-6 Opaque 0 3m38s foo-45 fubar-7 Opaque 0 3m38s foo-45 fubar-8 Opaque 0 3m38s foo-45 fubar-9 Opaque 0 3m38s foo-46 fubar-1 Opaque 0 3m39s foo-46 fubar-10 Opaque 0 3m38s foo-46 fubar-2 Opaque 0 3m39s foo-46 fubar-3 Opaque 0 3m39s foo-46 fubar-4 Opaque 0 3m39s foo-46 fubar-5 Opaque 0 3m38s foo-46 fubar-6 Opaque 0 3m38s foo-46 fubar-7 Opaque 0 3m38s foo-46 fubar-8 Opaque 0 3m38s foo-46 fubar-9 Opaque 0 3m38s foo-47 fubar-1 Opaque 0 3m38s foo-47 fubar-10 Opaque 0 3m37s foo-47 fubar-2 Opaque 0 3m38s foo-47 fubar-3 Opaque 0 3m38s foo-47 fubar-4 Opaque 0 3m38s foo-47 fubar-5 Opaque 0 3m37s foo-47 fubar-6 Opaque 0 3m37s foo-47 fubar-7 Opaque 0 3m37s foo-47 fubar-8 Opaque 0 3m37s foo-47 fubar-9 Opaque 0 3m37s foo-48 fubar-1 Opaque 0 3m38s foo-48 fubar-10 Opaque 0 3m37s foo-48 fubar-2 Opaque 0 3m38s foo-48 fubar-3 Opaque 0 3m38s foo-48 fubar-4 Opaque 0 3m38s foo-48 fubar-5 Opaque 0 3m37s foo-48 fubar-6 Opaque 0 3m37s foo-48 fubar-7 Opaque 0 3m37s foo-48 fubar-8 Opaque 0 3m37s foo-48 fubar-9 Opaque 0 3m37s foo-49 fubar-1 Opaque 0 3m39s foo-49 fubar-10 Opaque 0 3m38s foo-49 fubar-2 Opaque 0 3m39s foo-49 fubar-3 Opaque 0 3m39s foo-49 fubar-4 Opaque 0 3m39s foo-49 fubar-5 Opaque 0 3m39s foo-49 fubar-6 Opaque 0 3m39s foo-49 fubar-7 Opaque 0 3m38s foo-49 fubar-8 Opaque 0 3m38s foo-49 fubar-9 Opaque 0 3m38s foo-5 fubar-1 Opaque 0 3m49s foo-5 fubar-10 Opaque 0 3m48s foo-5 fubar-2 Opaque 0 3m49s foo-5 fubar-3 Opaque 0 3m49s foo-5 fubar-4 Opaque 0 3m49s foo-5 fubar-5 Opaque 0 3m49s foo-5 fubar-6 Opaque 0 3m49s foo-5 fubar-7 Opaque 0 3m49s foo-5 fubar-8 Opaque 0 3m49s foo-5 fubar-9 Opaque 0 3m48s foo-50 fubar-1 Opaque 0 3m39s foo-50 fubar-10 Opaque 0 3m38s foo-50 fubar-2 Opaque 0 3m39s foo-50 fubar-3 Opaque 0 3m39s foo-50 fubar-4 Opaque 0 3m39s foo-50 fubar-5 Opaque 0 3m39s foo-50 fubar-6 Opaque 0 3m38s foo-50 fubar-7 Opaque 0 3m38s foo-50 fubar-8 Opaque 0 3m38s foo-50 fubar-9 Opaque 0 3m38s foo-6 fubar-1 Opaque 0 3m50s foo-6 fubar-10 Opaque 0 3m49s foo-6 fubar-2 Opaque 0 3m49s foo-6 fubar-3 Opaque 0 3m49s foo-6 fubar-4 Opaque 0 3m49s foo-6 fubar-5 Opaque 0 3m49s foo-6 fubar-6 Opaque 0 3m49s foo-6 fubar-7 Opaque 0 3m49s foo-6 fubar-8 Opaque 0 3m49s foo-6 fubar-9 Opaque 0 3m49s foo-7 fubar-1 Opaque 0 3m50s foo-7 fubar-10 Opaque 0 3m49s foo-7 fubar-2 Opaque 0 3m49s foo-7 fubar-3 Opaque 0 3m49s foo-7 fubar-4 Opaque 0 3m49s foo-7 fubar-5 Opaque 0 3m49s foo-7 fubar-6 Opaque 0 3m49s foo-7 fubar-7 Opaque 0 3m49s foo-7 fubar-8 Opaque 0 3m49s foo-7 fubar-9 Opaque 0 3m49s foo-8 fubar-1 Opaque 0 3m50s foo-8 fubar-10 Opaque 0 3m49s foo-8 fubar-2 Opaque 0 3m49s foo-8 fubar-3 Opaque 0 3m49s foo-8 fubar-4 Opaque 0 3m49s foo-8 fubar-5 Opaque 0 3m49s foo-8 fubar-6 Opaque 0 3m49s foo-8 fubar-7 Opaque 0 3m49s foo-8 fubar-8 Opaque 0 3m49s foo-8 fubar-9 Opaque 0 3m49s foo-9 fubar-1 Opaque 0 3m48s foo-9 fubar-10 Opaque 0 3m48s foo-9 fubar-2 Opaque 0 3m48s foo-9 fubar-3 Opaque 0 3m48s foo-9 fubar-4 Opaque 0 3m48s foo-9 fubar-5 Opaque 0 3m48s foo-9 fubar-6 Opaque 0 3m48s foo-9 fubar-7 Opaque 0 3m48s foo-9 fubar-8 Opaque 0 3m48s foo-9 fubar-9 Opaque 0 3m48s % oc get secrets -A -l foo=bar | wc -l 193 % oc get secrets -n foo-1 NAME TYPE DATA AGE builder-dockercfg-s8rxp kubernetes.io/dockercfg 1 4m49s builder-token-qhv9s kubernetes.io/service-account-token 4 4m49s default-dockercfg-fzmbv kubernetes.io/dockercfg 1 4m49s default-token-jbpvv kubernetes.io/service-account-token 4 4m49s deployer-dockercfg-6sr2f kubernetes.io/dockercfg 1 4m49s deployer-token-twr5f kubernetes.io/service-account-token 4 4m49s fubar-1 Opaque 0 4m49s fubar-10 Opaque 0 4m48s fubar-2 Opaque 0 4m49s fubar-3 Opaque 0 4m49s fubar-4 Opaque 0 4m49s fubar-5 Opaque 0 4m49s fubar-6 Opaque 0 4m49s fubar-7 Opaque 0 4m49s fubar-8 Opaque 0 4m49s fubar-9 Opaque 0 4m48s % oc describe secret fubar-1 -n foo-1 Name: fubar-1 Namespace: foo-1 Labels: foo=bar Annotations: <none> Type: Opaque Data ====
the behaviour is totally weird oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=500' | jq '.items[].metadata.name' | wc 192 192 1939 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=501' | jq '.items[].metadata.name' | wc 191 191 1929 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=502' | jq '.items[].metadata.name' | wc 190 190 1919 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=503' | jq '.items[].metadata.name' | wc 190 190 1919 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=504' | jq '.items[].metadata.name' | wc 190 190 1919 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=204' | jq '.items[].metadata.name' | wc 204 204 2061 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=234' | jq '.items[].metadata.name' | wc 234 234 2364 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=534' | jq '.items[].metadata.name' | wc 170 170 1717 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=434' | jq '.items[].metadata.name' | wc 234 234 2363 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=434' | jq '.items[].metadata.name' | wc 234 234 2363 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=334' | jq '.items[].metadata.name' | wc 0 0 0 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=334' | jq '.items[].metadata.name' | wc 0 0 0 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=434' | jq '.items[].metadata.name' | wc 234 234 2363 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=4134' | jq '.items[].metadata.name' | wc 500 500 5050
it seems setting resourceVersion to 0 makes it work correctly oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=500&resourceVersion=0' | jq '.items[].metadata.name' | wc 500 500 5050 without resourceVersion set in the url oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=500' | jq '.items[].metadata.name' | wc 192 192 1939
watchcache doesn't support pagination - so the limit param is ignored with RV=0 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=50&resourceVersion=0' | jq '.items[].metadata.name' | wc 500 500 5050
ok, found the culprit https://github.com/openshift/kubernetes/pull/1303 reverting commit a2ad9f9e4aba6aae6657a3189bdced6dbc8ba4b5 recovers the same behaviour oc get secrets -A -l foo=bar | wc -l 501 now the question, why does this only affects Openshift?
it is present upstream too, Openshift just happen to have more objects by default. Working on a fix
This has to be backported to all stable branches https://github.com/openshift/kubernetes/pull/1303 however, the wrong behaviour is only exhibited since 1.24, because this change https://github.com/kubernetes/kubernetes/pull/108569 has triggered the problem.
I tried with 4.12 ci build(No one nightly build available). Did a quick test with attached script reproducer.sh, Steps as below, $ oc version Client Version: 4.11.0-fc.0 Kustomize Version: v4.5.4 Server Version: 4.12.0-0.ci-2022-07-01-060207 Kubernetes Version: v1.24.0-2362+d85aeef6706b52-dirty $ bash ./reproducer.sh ... namespace/foo-43 created secret/fubar-1 created secret/fubar-2 created secret/fubar-1 created secret/fubar-2 created $ oc get secrets -A -l foo=bar | wc -l 501 Listed all secrets with label, got the expected results, since the bug has label FastFix, no need wait nightly currently, so move the bug VERIFIED.Will re-test it when nightly build is available.
Retested with 4.12 nightly build, got the expected results, $ oc version Client Version: 4.11.0-fc.0 Kustomize Version: v4.5.4 Server Version: 4.12.0-0.nightly-2022-07-02-041854 Kubernetes Version: v1.24.0+52d428d $ bash ./reproducer.sh ... namespace/foo-43 created secret/fubar-1 created secret/fubar-2 created secret/fubar-1 created secret/fubar-2 created $ oc get secrets -A -l foo=bar | wc -l 501 kewang@kewang-mac ~/work/openshift/envmanual1 $
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399