Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: Got a lot of "Reconciler error" in cdi-deployment log after adding custom DataImportCron to hco Version-Release number of selected component (if applicable): CNV-v4.10.1-101 How reproducible: Always Steps to Reproduce: 1. Add custom DataImportCron to HCO eg: --------------8<------------------ spec: certConfig: ca: duration: 48h0m0s renewBefore: 24h0m0s server: duration: 24h0m0s renewBefore: 12h0m0s dataImportCronTemplates: - metadata: annotations: cdi.kubevirt.io/storage.bind.immediate.requested: "true" name: data-import-test2 spec: managedDataSource: data-source-test2 retentionPolicy: None schedule: '* * * * *' template: spec: source: registry: pullMethod: node url: docker://quay.io/containerdisks/fedora:35 storage: resources: requests: storage: 10Gi featureGates: enableCommonBootImageImport: true --------------8<------------------ Actual results: Got below error frequently till the import finished, log file is attached. $ oc logs cdi-deployment-7bbb74d84d-hjkjs -n openshift-cnv | grep Reconciler | grep data-source-test2 {"level":"error","ts":1652077998.9596689,"logger":"controller.dataimportcron-controller","msg":"Reconciler error","name":"data-source-test2","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datasources.cdi.kubevirt.io \"data-source-test2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078009.948743,"logger":"controller.datavolume-controller","msg":"Unable to update datavolume","name":"data-source-test2-3b3fc310abea","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"kubevirt.io/containerized-data-importer/pkg/controller.(*DatavolumeReconciler).reconcileDataVolumeStatus\n\t/remote-source/app/pkg/controller/datavolume-controller.go:2144\nkubevirt.io/containerized-data-importer/pkg/controller.(*DatavolumeReconciler).Reconcile\n\t/remote-source/app/pkg/controller/datavolume-controller.go:526\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078009.9488642,"logger":"controller.datavolume-controller","msg":"Reconciler error","name":"data-source-test2-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078010.129169,"logger":"controller.import-controller","msg":"Reconciler error","name":"data-source-test-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on persistentvolumeclaims \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078010.1938965,"logger":"controller.import-controller","msg":"Reconciler error","name":"data-source-test2-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on persistentvolumeclaims \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} $ oc get dv data-source-test2-3b3fc310abea NAME PHASE PROGRESS RESTARTS AGE data-source-test2-3b3fc310abea Succeeded 100.0% 18m Expected results: Better not to reduce the noise Additional info: log attached
Created attachment 1877988 [details] cdi deployment log
Oops, picked wrong version, corrected
Test on CNV-v4.10.3-4 The "Reconciler error" in cdi-deployment log has been reduced
Got 3 times "Reconciler error" now: {"level":"error","ts":1656315188.8290644,"logger":"controller.dataimportcron-controller","msg":"Reconciler error","name":"data-import-test1","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datasources.cdi.kubevirt.io \"data-source-test1\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1656315189.053071,"logger":"controller.dataimportcron-controller","msg":"Reconciler error","name":"data-import-test1","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on dataimportcrons.cdi.kubevirt.io \"data-import-test1\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1656315198.178084,"logger":"controller.datavolume-controller","msg":"Reconciler error","name":"data-source-test1-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test1-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 4.10.3 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:5675