Description of problem:

After upgrading from 4.8 to 4.9.11 "completed" pods are not deleted anymore, e.g: cronjobs seems to ignore the successfulJobsHistoryLimit value.

In the KCM there are some occurrences of these lines:

~~~
2022-01-10T09:00:08.627571256Z E0110 09:00:08.627527       1 shared_informer.go:243] unable to sync caches for garbage collector
2022-01-10T09:00:08.627571256Z E0110 09:00:08.627541       1 garbagecollector.go:242] timed out waiting for dependency graph builder sync during GC sync (attempt 5559)
~~~

and no webhooks seem to be blocking GC to run.

Version-Release number of selected component (if applicable):
OCP 4.9.11

How reproducible:
Always (in customer env)

Steps to Reproduce:
1. Create any object that will create pods in "Completed" state, e.g: a cronjob, and set the "successfulJobsHistoryLimit" parameter.


Actual results:

Pods in "Completed" status last forever, e.g:

~~~
 oc get cronjob cronjob-ldap-group-sync -o yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  creationTimestamp: "2021-08-02T12:09:55Z"
  generation: 1
  labels:
    template: cronjob-ldap-group-sync-secure
    template.openshift.io/template-instance-owner: c09af01c-4c10-11ea-810b-0a580a80002e
  name: cronjob-ldap-group-sync
  namespace: oe930-cron
  resourceVersion: "476752196"
  uid: 7c11ecf3-86f3-4a3f-bccc-47a37a2f9764
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      backoffLimit: 0
      template:
        metadata:
          creationTimestamp: null
        spec:
          activeDeadlineSeconds: 500
          containers:
          - command:
            - /bin/bash
            - -c
            - oc adm groups sync --confirm --sync-config=/ldap-sync/config/ldap-group-sync.yaml
              $([ -s /ldap-sync/config/whitelist.txt ] && echo --whitelist=/ldap-sync/config/whitelist.txt)
            image: registry.redhat.io/openshift4/ose-cli:latest
            imagePullPolicy: IfNotPresent
            name: cronjob-ldap-group-sync
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /ldap-sync/config
              name: ldap-sync-config
            - mountPath: /ldap-sync/ca
              name: ldap-sync-ca
            - mountPath: /ldap-sync/secrets
              name: ldap-bind-password
          dnsPolicy: ClusterFirst
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext: {}
          serviceAccount: ldap-group-syncer
          serviceAccountName: ldap-group-syncer
          terminationGracePeriodSeconds: 30
          volumes:
          - configMap:
              defaultMode: 420
              name: ldap-group-sync
            name: ldap-sync-config
          - configMap:
              defaultMode: 420
              name: ldap-group-sync-ca
            name: ldap-sync-ca
          - name: ldap-bind-password
            secret:
              defaultMode: 420
              secretName: ldap-bind-password
  schedule: '@hourly'
  successfulJobsHistoryLimit: 0
  suspend: false
status:
  lastScheduleTime: "2022-01-11T09:00:00Z"
  lastSuccessfulTime: "2022-01-10T15:02:26Z"
# oc get pods
NAME                                        READY   STATUS      RESTARTS   AGE
cronjob-ldap-group-sync-27363900--1-gns4s   0/1     Completed   0          16h
cronjob-ldap-group-sync-27363960--1-mcrpq   0/1     Completed   0          15h
cronjob-ldap-group-sync-27364020--1-ckprd   0/1     Completed   0          14h
cronjob-ldap-group-sync-27364080--1-mt4sn   0/1     Completed   0          13h
cronjob-ldap-group-sync-27364140--1-q29wf   0/1     Completed   0          12h
cronjob-ldap-group-sync-27364200--1-n6hcg   0/1     Completed   0          11h
cronjob-ldap-group-sync-27364260--1-4bqm4   0/1     Completed   0          10h
cronjob-ldap-group-sync-27364320--1-kbv8b   0/1     Completed   0          9h
cronjob-ldap-group-sync-27364380--1-gwd9b   0/1     Completed   0          8h
cronjob-ldap-group-sync-27364440--1-n4jrp   0/1     Completed   0          7h15m
cronjob-ldap-group-sync-27364500--1-v458d   0/1     Completed   0          6h15m
cronjob-ldap-group-sync-27364560--1-cp29v   0/1     Completed   0          5h15m
cronjob-ldap-group-sync-27364620--1-6cnxn   0/1     Completed   0          4h15m
cronjob-ldap-group-sync-27364680--1-h5f4n   0/1     Completed   0          3h15m
cronjob-ldap-group-sync-27364740--1-zwmsn   0/1     Completed   0          135m
cronjob-ldap-group-sync-27364800--1-2zkvf   0/1     Completed   0          75m
cronjob-ldap-group-sync-27364860--1-4fnv2   0/1     Completed   0          15m
~~~

Expected results:

Completed pods should be cleaned by GC after a while and particularly cronjobs pods with successfulJobsHistoryLimit set, should be honoured.

Additional info: