Bug 2223049

Description of problem:Overwritten/ReconcileHCO events are not consistently getting generated on overwriting many managed objects of HCO


Version-Release number of selected component (if applicable):
CNV-v4.14.0.rhel9-1238

How reproducible:
Intermittently reproducible

Steps to Reproduce:
1. Clean all existing events
2. Override a managed object, e.g. Role hyperconverged-cluster-operator-metrics 
3. Watch for events

Actual results:
No event was generated in HCO namespace
[cloud-user@ocp-ipi-executor-xl ~]$ oc get hco -n openshift-cnv kubevirt-hyperconverged -o json | jq ".status.conditions"
[
  {
    "lastTransitionTime": "2023-07-14T23:19:34Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 4,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "ReconcileComplete"
  },
  {
    "lastTransitionTime": "2023-07-14T23:19:37Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 4,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2023-07-14T23:19:37Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 4,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2023-07-14T23:19:37Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 4,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2023-07-14T23:19:37Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 4,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Upgradeable"
  }
]
[cloud-user@ocp-ipi-executor-xl ~]$ 
[cloud-user@ocp-ipi-executor-xl ~]$ oc get events -n openshift-cnv
No resources found in openshift-cnv namespace.
[cloud-user@ocp-ipi-executor-xl ~]$ 
I see the following entry in operator log:
========================
{"level":"error","ts":"2023-07-15T01:03:23Z","logger":"controller_hyperconverged","msg":"Failed to update HCO Status","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).logHyperConvergedUpdateError\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:637\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).updateHyperConverged\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:586\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:337\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2023-07-15T01:03:23Z","msg":"Reconciler error","controller":"hyperconverged-controller","object":{"name":"hco-controlled-cr-2a2c8e6e-4e58-4f0e-8eb3-04892f309d6c","namespace":"openshift-cnv"},"namespace":"openshift-cnv","name":"hco-controlled-cr-2a2c8e6e-4e58-4f0e-8eb3-04892f309d6c","reconcileID":"9f57b2de-6e12-40f8-93ef-2424129f5dcb","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2023-07-15T01:03:23Z","logger":"controller_hyperconverged","msg":"The reconciliation got triggered by a secondary CR object","Request.Namespace":"openshift-cnv","Request.Name":"hco-controlled-cr-2a2c8e6e-4e58-4f0e-8eb3-04892f309d6c"}

Attached is log collected.
Expected results:
Overwritten and ReconcileHCO events would be generated

Additional info:
Seen this for:
1) Role hyperconverged-cluster-operator-metrics
2) Service kubevirt-hyperconverged-operator-metrics
3) PriorityClass kubevirt-cluster-critical  -n openshift-cnv
4) KubeVirt kubevirt-kubevirt-hyperconverged

And more