Created attachment 1882186 [details]
CVO log file

Description of problem:

In a cluster, we can see CVO hotloops on the resources
$ grep -o 'Updating .*due to diff' cvo.log | sort | uniq -c
     32 Updating CRD performanceprofiles.performance.openshift.io due to diff
     32 Updating CronJob openshift-operator-lifecycle-manager/collect-profiles due to diff
     32 Updating OperatorGroup openshift-monitoring/openshift-cluster-monitoring due to diff
     32 Updating OperatorGroup openshift-operator-lifecycle-manager/olm-operators due to diff
     32 Updating OperatorGroup openshift-operators/global-operators due to diff
     32 Updating ValidatingWebhookConfiguration /performance-addon-operator due to diff

$ grep "Updating ValidatingWebhookConfiguration /performance-addon-operator due to diff" -A20 cvo.log

I0519 06:24:01.286135       1 generic.go:109] Updating ValidatingWebhookConfiguration /performance-addon-operator due to diff:   &unstructured.Unstructured{
  	Object: map[string]interface{}{
  		"apiVersion": string("admissionregistration.k8s.io/v1"),
  		"kind":       string("ValidatingWebhookConfiguration"),
  		"metadata":   map[string]interface{}{"annotations": map[string]interface{}{"include.release.openshift.io/ibm-cloud-managed": string("true"), "include.release.openshift.io/self-managed-high-availability": string("true"), "include.release.openshift.io/single-node-developer": string("true"), "service.beta.openshift.io/inject-cabundle": string("true")}, "creationTimestamp": string("2022-05-19T03:44:02Z"), "generation": int64(72), "managedFields": []interface{}{map[string]interface{}{"apiVersion": string("admissionregistration.k8s.io/v1"), "fieldsType": string("FieldsV1"), "fieldsV1": map[string]interface{}{"f:metadata": map[string]interface{}{"f:annotations": map[string]interface{}{".": map[string]interface{}{}, "f:include.release.openshift.io/ibm-cloud-managed": map[string]interface{}{}, "f:include.release.openshift.io/self-managed-high-availability": map[string]interface{}{}, "f:include.release.openshift.io/single-node-developer": map[string]interface{}{}, ...}, "f:ownerReferences": map[string]interface{}{".": map[string]interface{}{}, `k:{"uid":"5d28ef1d-78f0-4a88-9c2c-1f0fb01e3c57"}`: map[string]interface{}{}}}, "f:webhooks": map[string]interface{}{".": map[string]interface{}{}, `k:{"name":"vwb.performance.openshift.io"}`: map[string]interface{}{".": map[string]interface{}{}, "f:admissionReviewVersions": map[string]interface{}{}, "f:clientConfig": map[string]interface{}{".": map[string]interface{}{}, "f:service": map[string]interface{}{".": map[string]interface{}{}, "f:name": map[string]interface{}{}, "f:namespace": map[string]interface{}{}, "f:path": map[string]interface{}{}, ...}}, "f:failurePolicy": map[string]interface{}{}, ...}}}, "manager": string("cluster-version-operator"), ...}, map[string]interface{}{"apiVersion": string("admissionregistration.k8s.io/v1"), "fieldsType": string("FieldsV1"), "fieldsV1": map[string]interface{}{"f:webhooks": map[string]interface{}{`k:{"name":"vwb.performance.openshift.io"}`: map[string]interface{}{"f:clientConfig": map[string]interface{}{"f:caBundle": map[string]interface{}{}}}}}, "manager": string("Go-http-client"), ...}}, ...},
  		"webhooks": []interface{}{
  			map[string]interface{}{
  				"admissionReviewVersions": []interface{}{string("v1")},
  				"clientConfig": map[string]interface{}{
+ 					"caBundle": string("LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVVENDQWptZ0F3SUJBZ0lJQVBWdk9qMzhFNjB3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRZMU1qa3pNVGs0TlRBZQpGdzB5TWpBMU1Ua3dNelEyTWpSYUZ3MHlOREEzTVRjd016"...),
  					"service":  map[string]interface{}{"name": string("performance-addon-operator-service"), "namespace": string("openshift-cluster-node-tuning-operator"), "path": string("/validate-performance-openshift-io-v2-performanceprofile"), "port": int64(443)},
  				},
  				"failurePolicy":     string("Fail"),
  				"matchPolicy":       string("Equivalent"),
  				"name":              string("vwb.performance.openshift.io"),
+ 				"namespaceSelector": map[string]interface{}{},
+ 				"objectSelector":    map[string]interface{}{},
  				"rules":             []interface{}{map[string]interface{}{"apiGroups": []interface{}{string("performance.openshift.io")}, "apiVersions": []interface{}{string("v2")}, "operations": []interface{}{string("CREATE"), string("UPDATE")}, "resources": []interface{}{string("performanceprofiles")}, ...}},
  				"sideEffects":       string("None"),
  				"timeoutSeconds":    int64(10),
  			},
  		},
  	},
  }

$ cat manifests/0000_50_cluster-node-tuning-operator_45-webhook-configuration.yaml
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    include.release.openshift.io/ibm-cloud-managed: "true"
    service.beta.openshift.io/serving-cert-secret-name: performance-addon-operator-webhook-cert
  labels:
    name: performance-addon-operator-service
  name: performance-addon-operator-service
  namespace: openshift-cluster-node-tuning-operator
spec:
  ports:
    - name: "443"
      port: 443
      protocol: TCP
      targetPort: 4343
  selector:
    name: cluster-node-tuning-operator
  type: ClusterIP

---

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    include.release.openshift.io/ibm-cloud-managed: "true"
    service.beta.openshift.io/inject-cabundle: "true"
  name: performance-addon-operator
webhooks:
  - admissionReviewVersions:
      - v1
    clientConfig:
      service:
        name: performance-addon-operator-service
        namespace: openshift-cluster-node-tuning-operator
        path: /validate-performance-openshift-io-v2-performanceprofile
        port: 443
    failurePolicy: Fail
    matchPolicy: Equivalent
    name: vwb.performance.openshift.io
    rules:
      - apiGroups:
          - performance.openshift.io
        apiVersions:
          - v2
        operations:
          - CREATE
          - UPDATE
        resources:
          - performanceprofiles
        scope: '*'
    sideEffects: None
    timeoutSeconds: 10

Version-Release number of the following components:
4.11.0-0.nightly-2022-05-18-053037

How reproducible:
1/1

Steps to Reproduce:
1. Install a cluster
2.
3.

Actual results:
CVO hotloops on ValidatingWebhookConfiguration /performance-addon-operator

Expected results:
CVO doesn't hotloop on the resource.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag