Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2094012

Summary: Listing secrets in all namespaces with a specific labelSelector does not work properly
Product: OpenShift Container Platform Reporter: Raul Sevilla <rsevilla>
Component: kube-apiserverAssignee: Abu Kashem <akashem>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.11CC: akashem, mfojtik, nweinber, prubenda, qili, rsevilla, svetsa, wking, xxia
Target Milestone: ---Keywords: FastFix
Target Release: 4.12.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-17 19:49:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2103075    
Attachments:
Description Flags
Reproducer resource
none
Not kube-burner reproducer none

Description Raul Sevilla 2022-06-06 14:40:30 UTC 
Created attachment 1887205 [details]
Reproducer resource

Created attachment 1887205 [details]
Reproducer resource

Description of problem:

Listing secrets in all namespaces with a specific labelSelector does not work properly:

rsevilla@wonderland /tmp $ oc get secret -A -l kube-burner-job=cluster-density -v9
I0606 16:37:17.875754 2681352 loader.go:372] Config loaded from file:  /home/rsevilla/kubeconfig
I0606 16:37:17.884142 2681352 round_trippers.go:466] curl -v -XGET  -H "User-Agent: oc/4.10.0 (linux/amd64) kubernetes/04ad1b5" -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=kube-burner-job%3Dcluster-density&limit=500'
I0606 16:37:17.885481 2681352 round_trippers.go:495] HTTP Trace: DNS Lookup for api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com resolved to [{34.208.8.2 } {44.224.254.41 } {44.242.34.69 } {54.218.246.157 }]
I0606 16:37:18.094176 2681352 round_trippers.go:510] HTTP Trace: Dial to tcp:34.208.8.2:6443 succeed
I0606 16:37:18.580598 2681352 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 208 ms TLSHandshake 214 ms ServerProcessing 271 ms Duration 696 ms
I0606 16:37:18.580656 2681352 round_trippers.go:577] Response Headers:
I0606 16:37:18.580682 2681352 round_trippers.go:580]     Cache-Control: no-cache, private
I0606 16:37:18.580703 2681352 round_trippers.go:580]     Content-Type: application/json
I0606 16:37:18.580722 2681352 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 7eb65202-ec98-465b-b7d4-eea85843f875
I0606 16:37:18.580742 2681352 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 64cc9fb8-ee32-4d93-bb3f-f10697c70027
I0606 16:37:18.580761 2681352 round_trippers.go:580]     Content-Length: 1610
I0606 16:37:18.580779 2681352 round_trippers.go:580]     Date: Mon, 06 Jun 2022 14:37:18 GMT
I0606 16:37:18.580798 2681352 round_trippers.go:580]     Audit-Id: ec6343eb-5924-4f28-aeb7-2d67fa181467
I0606 16:37:18.580893 2681352 request.go:1181] Response Body: {"kind":"Table","apiVersion":"meta.k8s.io/v1","metadata":{"resourceVersion":"1154448"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names","priority":0},{"name":"Type","type":"string","format":"","description":"Used to facilitate programmatic handling of secret data. More info: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types","priority":0},{"name":"Data","type":"string","format":"","description":"Data contains the secret data. Each key must consist of alphanumeric characters, '-', '_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in https://tools.ietf.org/html/rfc4648#section-4","priority":0},{"name":"Age","type":"string","format":"","description":"CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.\n\nPopulated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata","priority":0}],"rows":[]}
No resources found

rsevilla@wonderland /tmp $ ./oc get secret --no-headers  -l kube-burner-job=cluster-density -n cluster-density-1 --show-labels 
cluster-density-1    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-10   Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-2    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-3    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-4    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-5    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-6    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-7    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-8    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-9    Opaque   1     45m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4


In the cluster above there were 100 secrets created across 100 namespaces, all of them sharing the label kube-burner-job=cluster-density

On the other hand, listing configmaps, (which have similar labels worked fine) as shown below

rsevilla@wonderland /tmp $ ./oc get cm --no-headers -A -l kube-burner-job=cluster-density  | wc -l
100



Version-Release number of selected component (if applicable):

rsevilla@wonderland /tmp $ ./oc version
Client Version: 4.11.0-0.nightly-2022-06-04-014713
Kustomize Version: v4.5.4
Server Version: 4.11.0-0.nightly-2022-06-04-014713
Kubernetes Version: v1.24.0+bb9c2f1



Steps to Reproduce:



1. $ for N in {1..10}; do export N ; envsubst < ns.yml | oc apply -f - & done (ns.yml is attached to this BZ)
2. $ oc get secret -A -l foo=bar | wc -l
0


Expected results:

Oc lists all objects


Additional info:


After deleting 9 namespaces out of 10, the listing command works

rsevilla@wonderland /tmp $ ./oc delete ns cluster-density-{2..10}                                                                                                                                                                               
namespace "cluster-density-2" deleted                                                                                                                                                                                                         
namespace "cluster-density-3" deleted                                                                                                                                                                                                         
namespace "cluster-density-4" deleted                                                                                                                                                                                                         
namespace "cluster-density-5" deleted                                                                                                                                                                                                         
namespace "cluster-density-6" deleted                                                                                                                                                                                                         
namespace "cluster-density-7" deleted                                                                                                                                                                                                         
namespace "cluster-density-8" deleted                                                                                                                                                                                                         
namespace "cluster-density-9" deleted                     
namespace "cluster-density-10" deleted
rsevilla@wonderland /tmp $ ./oc get secret --no-headers  -l kube-burner-job=cluster-density -n cluster-density-1 --show-labels  
cluster-density-1    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-10   Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-2    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-3    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-4    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-5    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-6    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-7    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-8    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
cluster-density-9    Opaque   1     50m   kube-burner-index=5,kube-burner-job=cluster-density,kube-burner-uuid=689a523d-4c9d-48a6-9e2d-2dddfb8059b4
Comment 1 Antonio Ojea 2022-06-06 15:26:05 UTC 
I can't reproduce the problem in vanilla kubernetes, it seems the limit variable in openshift behaves differently.

A more generic reproducder, 

1. Create 100 fake secrets

$ cat secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  generateName: mysecret-
  labels:
    test: test
type: Opaque
data:

$ oc create namespace testns

$ for i in `seq 1 100` ; do oc create -f secret.yaml -n testns; done

2. Try to list the secrets in all namespaces filtering by the label used, with the default limit variable = 500

oc get --raw 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=test%3Dtest&limit=500' | jq .items[].metadata.name | wc
      0       0       0

---> No resources

3. List the secrets without limit or with a higher limit value

oc get --raw 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=test%3Dtest&limit=687' | jq .items[].metadata.name | wc
    100     100    1700

oc get --raw 'https://api.ci-4.11-aws-sdn-control-plane.perfscale.devcluster.openshift.com:6443/api/v1/secrets?labelSelector=test%3Dtest' | jq .items[].metadata.name | wc
    100     100    1700
Comment 6 Raul Sevilla 2022-06-28 15:25:54 UTC 
Created attachment 1893215 [details]
Not kube-burner reproducer
Comment 7 Raul Sevilla 2022-06-28 15:26:25 UTC 
(In reply to Antonio Ojea from comment #5)
> To be clear, I'm not able to reproduce it in a cluster where kube-burner
> didn't run
> 
> Next step is to reproduce it without kube-burner
> 
> 1. Create 100 fake secrets
> 
> $ cat secret.yaml 
> apiVersion: v1
> kind: Secret
> metadata:
>   generateName: mysecret-
>   labels:
>     test: test
> type: Opaque
> data:
> 
> $ oc create namespace testns
> 
> $ for i in `seq 1 100` ; do oc create -f secret.yaml -n testns; done
> 
> 2. Try to list the secrets in all namespaces filtering by the label used,
> with the default limit variable = 500
> 
> oc get secrets -A | wc -l
> 
> 
> 
> if this is not reproducible we have to  understand what are the differences
 (
Hey!, I did reproduce this issue w/o kube-burner using the reproducer I just attached (reproducer.sh), that creates 50 namespaces with 10 secrets labeled with foo=bar each:
rsevilla@wonderland /tmp $ ./reproducer.sh > /dev/null
rsevilla@wonderland /tmp $ oc get secret -A -l foo=bar | wc -l
79
rsevilla@wonderland /tmp $ kubectl get secret -A -l foo=bar | wc -l
79
rsevilla@wonderland /tmp $ oc version
kubeClient Version: 4.10.15
Server Version: 4.11.0-0.nightly-2022-06-15-222801
Kubernetes Version: v1.24.0+25f9057
rsevilla@wonderland /tmp $ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.24.0
Kustomize Version: v4.5.4
Server Version: v1.24.0+25f9057
Comment 8 Paige Rubendall 2022-06-28 15:48:09 UTC 
Seeing the same thing with a newer client and server versions

 % oc version
Client Version: 4.11.0-0.nightly-2022-06-21-094850
Kustomize Version: v4.5.4
Server Version: 4.11.0-0.nightly-2022-06-25-132614
Kubernetes Version: v1.24.0+9ddc8b1


 % kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.24.1
Kustomize Version: v4.5.4
Server Version: v1.24.0+9ddc8b1

% oc get secrets -A -l foo=bar          
NAMESPACE   NAME       TYPE     DATA   AGE
foo-37      fubar-8    Opaque   0      3m41s
foo-37      fubar-9    Opaque   0      3m40s
foo-38      fubar-1    Opaque   0      3m41s
foo-38      fubar-10   Opaque   0      3m40s
foo-38      fubar-2    Opaque   0      3m41s
foo-38      fubar-3    Opaque   0      3m41s
foo-38      fubar-4    Opaque   0      3m41s
foo-38      fubar-5    Opaque   0      3m41s
foo-38      fubar-6    Opaque   0      3m41s
foo-38      fubar-7    Opaque   0      3m41s
foo-38      fubar-8    Opaque   0      3m40s
foo-38      fubar-9    Opaque   0      3m40s
foo-39      fubar-1    Opaque   0      3m41s
foo-39      fubar-10   Opaque   0      3m40s
foo-39      fubar-2    Opaque   0      3m41s
foo-39      fubar-3    Opaque   0      3m41s
foo-39      fubar-4    Opaque   0      3m41s
foo-39      fubar-5    Opaque   0      3m41s
foo-39      fubar-6    Opaque   0      3m41s
foo-39      fubar-7    Opaque   0      3m41s
foo-39      fubar-8    Opaque   0      3m40s
foo-39      fubar-9    Opaque   0      3m40s
foo-4       fubar-1    Opaque   0      3m49s
foo-4       fubar-10   Opaque   0      3m48s
foo-4       fubar-2    Opaque   0      3m49s
foo-4       fubar-3    Opaque   0      3m49s
foo-4       fubar-4    Opaque   0      3m49s
foo-4       fubar-5    Opaque   0      3m49s
foo-4       fubar-6    Opaque   0      3m49s
foo-4       fubar-7    Opaque   0      3m49s
foo-4       fubar-8    Opaque   0      3m49s
foo-4       fubar-9    Opaque   0      3m48s
foo-40      fubar-1    Opaque   0      3m41s
foo-40      fubar-10   Opaque   0      3m40s
foo-40      fubar-2    Opaque   0      3m41s
foo-40      fubar-3    Opaque   0      3m41s
foo-40      fubar-4    Opaque   0      3m41s
foo-40      fubar-5    Opaque   0      3m40s
foo-40      fubar-6    Opaque   0      3m40s
foo-40      fubar-7    Opaque   0      3m40s
foo-40      fubar-8    Opaque   0      3m40s
foo-40      fubar-9    Opaque   0      3m40s
foo-41      fubar-1    Opaque   0      3m39s
foo-41      fubar-10   Opaque   0      3m39s
foo-41      fubar-2    Opaque   0      3m39s
foo-41      fubar-3    Opaque   0      3m39s
foo-41      fubar-4    Opaque   0      3m39s
foo-41      fubar-5    Opaque   0      3m39s
foo-41      fubar-6    Opaque   0      3m39s
foo-41      fubar-7    Opaque   0      3m39s
foo-41      fubar-8    Opaque   0      3m39s
foo-41      fubar-9    Opaque   0      3m39s
foo-42      fubar-1    Opaque   0      3m39s
foo-42      fubar-10   Opaque   0      3m38s
foo-42      fubar-2    Opaque   0      3m39s
foo-42      fubar-3    Opaque   0      3m39s
foo-42      fubar-4    Opaque   0      3m39s
foo-42      fubar-5    Opaque   0      3m39s
foo-42      fubar-6    Opaque   0      3m39s
foo-42      fubar-7    Opaque   0      3m38s
foo-42      fubar-8    Opaque   0      3m38s
foo-42      fubar-9    Opaque   0      3m38s
foo-43      fubar-1    Opaque   0      3m40s
foo-43      fubar-10   Opaque   0      3m39s
foo-43      fubar-2    Opaque   0      3m39s
foo-43      fubar-3    Opaque   0      3m39s
foo-43      fubar-4    Opaque   0      3m39s
foo-43      fubar-5    Opaque   0      3m39s
foo-43      fubar-6    Opaque   0      3m39s
foo-43      fubar-7    Opaque   0      3m39s
foo-43      fubar-8    Opaque   0      3m39s
foo-43      fubar-9    Opaque   0      3m39s
foo-44      fubar-1    Opaque   0      3m39s
foo-44      fubar-10   Opaque   0      3m38s
foo-44      fubar-2    Opaque   0      3m39s
foo-44      fubar-3    Opaque   0      3m38s
foo-44      fubar-4    Opaque   0      3m38s
foo-44      fubar-5    Opaque   0      3m38s
foo-44      fubar-6    Opaque   0      3m38s
foo-44      fubar-7    Opaque   0      3m38s
foo-44      fubar-8    Opaque   0      3m38s
foo-44      fubar-9    Opaque   0      3m38s
foo-45      fubar-1    Opaque   0      3m39s
foo-45      fubar-10   Opaque   0      3m38s
foo-45      fubar-2    Opaque   0      3m39s
foo-45      fubar-3    Opaque   0      3m39s
foo-45      fubar-4    Opaque   0      3m39s
foo-45      fubar-5    Opaque   0      3m38s
foo-45      fubar-6    Opaque   0      3m38s
foo-45      fubar-7    Opaque   0      3m38s
foo-45      fubar-8    Opaque   0      3m38s
foo-45      fubar-9    Opaque   0      3m38s
foo-46      fubar-1    Opaque   0      3m39s
foo-46      fubar-10   Opaque   0      3m38s
foo-46      fubar-2    Opaque   0      3m39s
foo-46      fubar-3    Opaque   0      3m39s
foo-46      fubar-4    Opaque   0      3m39s
foo-46      fubar-5    Opaque   0      3m38s
foo-46      fubar-6    Opaque   0      3m38s
foo-46      fubar-7    Opaque   0      3m38s
foo-46      fubar-8    Opaque   0      3m38s
foo-46      fubar-9    Opaque   0      3m38s
foo-47      fubar-1    Opaque   0      3m38s
foo-47      fubar-10   Opaque   0      3m37s
foo-47      fubar-2    Opaque   0      3m38s
foo-47      fubar-3    Opaque   0      3m38s
foo-47      fubar-4    Opaque   0      3m38s
foo-47      fubar-5    Opaque   0      3m37s
foo-47      fubar-6    Opaque   0      3m37s
foo-47      fubar-7    Opaque   0      3m37s
foo-47      fubar-8    Opaque   0      3m37s
foo-47      fubar-9    Opaque   0      3m37s
foo-48      fubar-1    Opaque   0      3m38s
foo-48      fubar-10   Opaque   0      3m37s
foo-48      fubar-2    Opaque   0      3m38s
foo-48      fubar-3    Opaque   0      3m38s
foo-48      fubar-4    Opaque   0      3m38s
foo-48      fubar-5    Opaque   0      3m37s
foo-48      fubar-6    Opaque   0      3m37s
foo-48      fubar-7    Opaque   0      3m37s
foo-48      fubar-8    Opaque   0      3m37s
foo-48      fubar-9    Opaque   0      3m37s
foo-49      fubar-1    Opaque   0      3m39s
foo-49      fubar-10   Opaque   0      3m38s
foo-49      fubar-2    Opaque   0      3m39s
foo-49      fubar-3    Opaque   0      3m39s
foo-49      fubar-4    Opaque   0      3m39s
foo-49      fubar-5    Opaque   0      3m39s
foo-49      fubar-6    Opaque   0      3m39s
foo-49      fubar-7    Opaque   0      3m38s
foo-49      fubar-8    Opaque   0      3m38s
foo-49      fubar-9    Opaque   0      3m38s
foo-5       fubar-1    Opaque   0      3m49s
foo-5       fubar-10   Opaque   0      3m48s
foo-5       fubar-2    Opaque   0      3m49s
foo-5       fubar-3    Opaque   0      3m49s
foo-5       fubar-4    Opaque   0      3m49s
foo-5       fubar-5    Opaque   0      3m49s
foo-5       fubar-6    Opaque   0      3m49s
foo-5       fubar-7    Opaque   0      3m49s
foo-5       fubar-8    Opaque   0      3m49s
foo-5       fubar-9    Opaque   0      3m48s
foo-50      fubar-1    Opaque   0      3m39s
foo-50      fubar-10   Opaque   0      3m38s
foo-50      fubar-2    Opaque   0      3m39s
foo-50      fubar-3    Opaque   0      3m39s
foo-50      fubar-4    Opaque   0      3m39s
foo-50      fubar-5    Opaque   0      3m39s
foo-50      fubar-6    Opaque   0      3m38s
foo-50      fubar-7    Opaque   0      3m38s
foo-50      fubar-8    Opaque   0      3m38s
foo-50      fubar-9    Opaque   0      3m38s
foo-6       fubar-1    Opaque   0      3m50s
foo-6       fubar-10   Opaque   0      3m49s
foo-6       fubar-2    Opaque   0      3m49s
foo-6       fubar-3    Opaque   0      3m49s
foo-6       fubar-4    Opaque   0      3m49s
foo-6       fubar-5    Opaque   0      3m49s
foo-6       fubar-6    Opaque   0      3m49s
foo-6       fubar-7    Opaque   0      3m49s
foo-6       fubar-8    Opaque   0      3m49s
foo-6       fubar-9    Opaque   0      3m49s
foo-7       fubar-1    Opaque   0      3m50s
foo-7       fubar-10   Opaque   0      3m49s
foo-7       fubar-2    Opaque   0      3m49s
foo-7       fubar-3    Opaque   0      3m49s
foo-7       fubar-4    Opaque   0      3m49s
foo-7       fubar-5    Opaque   0      3m49s
foo-7       fubar-6    Opaque   0      3m49s
foo-7       fubar-7    Opaque   0      3m49s
foo-7       fubar-8    Opaque   0      3m49s
foo-7       fubar-9    Opaque   0      3m49s
foo-8       fubar-1    Opaque   0      3m50s
foo-8       fubar-10   Opaque   0      3m49s
foo-8       fubar-2    Opaque   0      3m49s
foo-8       fubar-3    Opaque   0      3m49s
foo-8       fubar-4    Opaque   0      3m49s
foo-8       fubar-5    Opaque   0      3m49s
foo-8       fubar-6    Opaque   0      3m49s
foo-8       fubar-7    Opaque   0      3m49s
foo-8       fubar-8    Opaque   0      3m49s
foo-8       fubar-9    Opaque   0      3m49s
foo-9       fubar-1    Opaque   0      3m48s
foo-9       fubar-10   Opaque   0      3m48s
foo-9       fubar-2    Opaque   0      3m48s
foo-9       fubar-3    Opaque   0      3m48s
foo-9       fubar-4    Opaque   0      3m48s
foo-9       fubar-5    Opaque   0      3m48s
foo-9       fubar-6    Opaque   0      3m48s
foo-9       fubar-7    Opaque   0      3m48s
foo-9       fubar-8    Opaque   0      3m48s
foo-9       fubar-9    Opaque   0      3m48s
 % oc get secrets -A -l foo=bar | wc -l 
     193


 % oc get secrets -n foo-1
NAME                       TYPE                                  DATA   AGE
builder-dockercfg-s8rxp    kubernetes.io/dockercfg               1      4m49s
builder-token-qhv9s        kubernetes.io/service-account-token   4      4m49s
default-dockercfg-fzmbv    kubernetes.io/dockercfg               1      4m49s
default-token-jbpvv        kubernetes.io/service-account-token   4      4m49s
deployer-dockercfg-6sr2f   kubernetes.io/dockercfg               1      4m49s
deployer-token-twr5f       kubernetes.io/service-account-token   4      4m49s
fubar-1                    Opaque                                0      4m49s
fubar-10                   Opaque                                0      4m48s
fubar-2                    Opaque                                0      4m49s
fubar-3                    Opaque                                0      4m49s
fubar-4                    Opaque                                0      4m49s
fubar-5                    Opaque                                0      4m49s
fubar-6                    Opaque                                0      4m49s
fubar-7                    Opaque                                0      4m49s
fubar-8                    Opaque                                0      4m49s
fubar-9                    Opaque                                0      4m48s


 % oc describe secret fubar-1  -n foo-1
Name:         fubar-1
Namespace:    foo-1
Labels:       foo=bar
Annotations:  <none>

Type:  Opaque

Data
====
Comment 11 Antonio Ojea 2022-06-29 09:57:53 UTC 
the behaviour is totally weird

 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=500' | jq '.items[].metadata.name' | wc
    192     192    1939
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=501' | jq '.items[].metadata.name' | wc
    191     191    1929
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=502' | jq '.items[].metadata.name' | wc
    190     190    1919
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=503' | jq '.items[].metadata.name' | wc
    190     190    1919
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=504' | jq '.items[].metadata.name' | wc
    190     190    1919
 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=204' | jq '.items[].metadata.name' | wc
    204     204    2061
 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=234' | jq '.items[].metadata.name' | wc
    234     234    2364
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=534' | jq '.items[].metadata.name' | wc
    170     170    1717
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=434' | jq '.items[].metadata.name' | wc
    234     234    2363
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=434' | jq '.items[].metadata.name' | wc
    234     234    2363
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=334' | jq '.items[].metadata.name' | wc
      0       0       0
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=334' | jq '.items[].metadata.name' | wc
      0       0       0
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=434' | jq '.items[].metadata.name' | wc
    234     234    2363
oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=4134' | jq '.items[].metadata.name' | wc
    500     500    5050
Comment 12 Antonio Ojea 2022-06-29 10:10:45 UTC 
it seems setting resourceVersion to 0 makes it work correctly

oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=500&resourceVersion=0' | jq '.items[].metadata.name' | wc
    500     500    5050

without resourceVersion set in the url

 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=500' | jq '.items[].metadata.name' | wc
    192     192    1939
Comment 13 Antonio Ojea 2022-06-29 10:19:35 UTC 

watchcache doesn't support pagination - so the limit param is ignored with RV=0

 oc get --raw 'https://api.ci-ln-bnqrb3t-72292.origin-ci-int-gce.dev.rhcloud.com:6443/api/v1/secrets?labelSelector=foo%3Dbar&limit=50&resourceVersion=0' | jq '.items[].metadata.name' | wc
    500     500    5050
Comment 14 Antonio Ojea 2022-06-29 13:15:49 UTC 
ok, found the culprit https://github.com/openshift/kubernetes/pull/1303

reverting commit a2ad9f9e4aba6aae6657a3189bdced6dbc8ba4b5 recovers the same behaviour

oc get secrets -A -l foo=bar | wc -l 
501


now the question, why does this only affects Openshift?
Comment 15 Antonio Ojea 2022-06-29 14:07:10 UTC 
it is present upstream too, Openshift just happen to have more objects by default.

Working on a fix
Comment 16 Antonio Ojea 2022-06-30 08:18:04 UTC 
This has to be backported to all stable branches https://github.com/openshift/kubernetes/pull/1303

however, the wrong behaviour is only exhibited since 1.24, because this change https://github.com/kubernetes/kubernetes/pull/108569 has triggered  the problem.
Comment 19 Ke Wang 2022-07-01 10:34:49 UTC 
I tried with 4.12 ci build(No one nightly build available). Did a quick test with attached script reproducer.sh, 

Steps as below,
$ oc version
Client Version: 4.11.0-fc.0
Kustomize Version: v4.5.4
Server Version: 4.12.0-0.ci-2022-07-01-060207
Kubernetes Version: v1.24.0-2362+d85aeef6706b52-dirty

$ bash ./reproducer.sh
...
namespace/foo-43 created
secret/fubar-1 created
secret/fubar-2 created
secret/fubar-1 created
secret/fubar-2 created

 $ oc get secrets -A -l foo=bar | wc -l
     501

Listed all secrets with label, got the expected results, since the bug has label FastFix, no need wait nightly currently, so move the bug VERIFIED.Will re-test it when nightly  build is available.
Comment 20 Ke Wang 2022-07-04 04:02:14 UTC 
Retested with 4.12 nightly build, got the expected results,

$ oc version
Client Version: 4.11.0-fc.0
Kustomize Version: v4.5.4
Server Version: 4.12.0-0.nightly-2022-07-02-041854
Kubernetes Version: v1.24.0+52d428d

$ bash ./reproducer.sh
...
namespace/foo-43 created
secret/fubar-1 created
secret/fubar-2 created
secret/fubar-1 created
secret/fubar-2 created

$ oc get secrets -A -l foo=bar | wc -l
     501
 kewang@kewang-mac ~/work/openshift/envmanual1 $
Comment 25 errata-xmlrpc 2023-01-17 19:49:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399