Bug 2046429

Summary: CDI operator gets stuck on product removal
Product: Container Native Virtualization (CNV) Reporter: Simone Tiraboschi <stirabos>
Component: StorageAssignee: Arnon Gilboa <agilboa>
Status: CLOSED DUPLICATE QA Contact: Natalie Gavrielov <ngavrilo>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.10.0CC: agilboa, cnv-qe-bugs
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: CNV v4.10.0-635 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-02-03 15:04:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
CDI operator logs
none
CDI operator logs v1.44.0 none

Description Simone Tiraboschi 2022-01-26 16:43:23 UTC 
Description of problem:
Probably as a side effect of the fix for:
https://bugzilla.redhat.com/show_bug.cgi?id=2037312

now CDI operator deletes DataImportCron CRD early in the game,
but then it fails later on in the deletion process because DataImportCron CRD is not there:

2022-01-26T15:45:54.154258146Z {"level":"error","ts":1643211954.1541657,"logger":"cdi-operator","msg":"error invoking callback for","type":"*v1.Deployment","error":"the server could not find the requested resource (get dataimportcrons.cdi.kubevirt.io)","stacktrace":"kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/callbacks.(*CallbackDispatcher).InvokeCallbacks
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/callbacks/callback-dispatcher.go:157
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).InvokeCallbacks
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:549
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).InvokeDeleteCallbacks
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:515
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).ReconcileDelete
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:713
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).Reconcile
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:136
kubevirt.io/containerized-data-importer/pkg/operator/controller.(*ReconcileCDI).Reconcile
	pkg/operator/controller/controller.go:233
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

Version-Release number of selected component (if applicable):
CDI v1.43.1

How reproducible:
100%

Steps to Reproduce:
1. deploy CNV
2. try to install it
3.

Actual results:
CDI CR is marked for deletion but protected by a finalized controlled by CDI operator:

apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
  annotations:
    cdi.kubevirt.io/configAuthority: ""
  creationTimestamp: "2022-01-26T15:24:13Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2022-01-26T15:35:40Z"
  finalizers:
  - operator.cdi.kubevirt.io

DataImportCron CRD is not there anymore

CDI operator is looping forever on:
{"level":"error","ts":1643211954.1541657,"logger":"cdi-operator","msg":"error invoking callback for","type":"*v1.Deployment","error":"the server could not find the requested resource (get dataimportcrons.cdi.kubevirt.io)","stacktrace":"kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/callbacks.(*CallbackDispatcher).InvokeCallbacks

Expected results:
We can successfully remove the product

Additional info:
I think that let the OLM manage the CRDs is a better and safer idea
Comment 2 Simone Tiraboschi 2022-02-02 08:15:11 UTC 
With CDI v1.44.0 which includes https://github.com/kubevirt/containerized-data-importer/pull/2129 now we see a slightly different pattern but basically the bug is still there:

CDI operator is looping forever on

2022-02-01T16:07:40.308086379Z {"level":"error","ts":1643731660.3080544,"logger":"cdi-operator","msg":"error getting apiserver ca bundle","error":"ConfigMap \"cdi-apiserver-signer-bundle\" not found","stacktrace":"kubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.getAPIServerCABundle\n\tpkg/operator/resources/cluster/apiserver.go:542\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createDataImportCronValidatingWebhook\n\tpkg/operator/resources/cluster/apiserver.go:244\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createDynamicAPIServerResources\n\tpkg/operator/resources/cluster/apiserver.go:57\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createResourceGroup\n\tpkg/operator/resources/cluster/factory.go:102\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createAllResources\n\tpkg/operator/resources/cluster/factory.go:88\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.CreateAllDynamicResources\n\tpkg/operator/resources/cluster/factory.go:77\nkubevirt.io/containerized-data-importer/pkg/operator/controller.(*ReconcileCDI).GetAllResources\n\tpkg/operator/controller/cr-manager.go:126\nkubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).InvokeDeleteCallbacks\n\tvendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:509\nkubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).ReconcileDelete\n\tvendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:713\nkubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).Reconcile\n\tvendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:136\nkubevirt.io/containerized-data-importer/pkg/operator/controller.(*ReconcileCDI).Reconcile\n\tpkg/operator/controller/controller.go:233\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
Comment 3 Simone Tiraboschi 2022-02-02 08:16:10 UTC 
Created attachment 1858560 [details]
CDI operator logs v1.44.0
Comment 4 Natalie Gavrielov 2022-02-03 11:43:07 UTC 
Hi Arnon, can you please have a look, is it the same thing (comment 2)
Comment 5 Arnon Gilboa 2022-02-03 15:04:33 UTC 
I see no reason to keep both bzs, closing this one as dup.

*** This bug has been marked as a duplicate of bug 2037312 ***