Bug 2091982

+++ This bug was initially created as a clone of Bug #2083039 +++

Description of problem:
Got a lot of "Reconciler error" in cdi-deployment log after adding custom DataImportCron to hco

Version-Release number of selected component (if applicable):
CNV-v4.10.1-101

How reproducible:
Always

Steps to Reproduce:
1. Add custom DataImportCron to HCO
eg:
--------------8<------------------
  spec:
    certConfig:
      ca:
        duration: 48h0m0s
        renewBefore: 24h0m0s
      server:
        duration: 24h0m0s
        renewBefore: 12h0m0s
    dataImportCronTemplates:
    - metadata:
        annotations:
          cdi.kubevirt.io/storage.bind.immediate.requested: "true"
        name: data-import-test2
      spec:
        managedDataSource: data-source-test2
        retentionPolicy: None
        schedule: '* * * * *'
        template:
          spec:
            source:
              registry:
                pullMethod: node
                url: docker://quay.io/containerdisks/fedora:35
            storage:
              resources:
                requests:
                  storage: 10Gi
    featureGates:
      enableCommonBootImageImport: true
--------------8<------------------


Actual results:
Got below error frequently till the import finished, log file is attached.

$ oc logs cdi-deployment-7bbb74d84d-hjkjs  -n openshift-cnv | grep Reconciler  | grep data-source-test2 
{"level":"error","ts":1652077998.9596689,"logger":"controller.dataimportcron-controller","msg":"Reconciler error","name":"data-source-test2","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datasources.cdi.kubevirt.io \"data-source-test2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1652078009.948743,"logger":"controller.datavolume-controller","msg":"Unable to update datavolume","name":"data-source-test2-3b3fc310abea","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"kubevirt.io/containerized-data-importer/pkg/controller.(*DatavolumeReconciler).reconcileDataVolumeStatus\n\t/remote-source/app/pkg/controller/datavolume-controller.go:2144\nkubevirt.io/containerized-data-importer/pkg/controller.(*DatavolumeReconciler).Reconcile\n\t/remote-source/app/pkg/controller/datavolume-controller.go:526\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1652078009.9488642,"logger":"controller.datavolume-controller","msg":"Reconciler error","name":"data-source-test2-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1652078010.129169,"logger":"controller.import-controller","msg":"Reconciler error","name":"data-source-test-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on persistentvolumeclaims \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1652078010.1938965,"logger":"controller.import-controller","msg":"Reconciler error","name":"data-source-test2-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on persistentvolumeclaims \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}


$ oc get dv data-source-test2-3b3fc310abea 
NAME                             PHASE       PROGRESS   RESTARTS   AGE
data-source-test2-3b3fc310abea   Succeeded   100.0%                18m

Expected results:
Better not to reduce the noise


Additional info:
log attached

--- Additional comment from Yan Du on 2022-05-09 07:02:06 UTC ---