Bug 2046429 - CDI operator gets stuck on product removal
Summary: CDI operator gets stuck on product removal
Keywords:
Status: CLOSED DUPLICATE of bug 2037312
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.10.0
Assignee: Arnon Gilboa
QA Contact: Natalie Gavrielov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-26 16:43 UTC by Simone Tiraboschi
Modified: 2022-02-03 15:04 UTC (History)
2 users (show)

Fixed In Version: CNV v4.10.0-635
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-03 15:04:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
CDI operator logs (17.00 KB, application/gzip)
2022-01-26 16:48 UTC, Simone Tiraboschi
no flags Details
CDI operator logs v1.44.0 (22.09 KB, application/gzip)
2022-02-02 08:16 UTC, Simone Tiraboschi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt containerized-data-importer pull 2129 0 None Merged Fix CRDs deletion in operator deletion 2022-02-02 08:10:04 UTC
Github kubevirt containerized-data-importer pull 2130 0 None Merged [release-v1.43] Fix CRDs deletion in operator deletion 2022-02-02 08:10:09 UTC

Description Simone Tiraboschi 2022-01-26 16:43:23 UTC
Description of problem:
Probably as a side effect of the fix for:
https://bugzilla.redhat.com/show_bug.cgi?id=2037312

now CDI operator deletes DataImportCron CRD early in the game,
but then it fails later on in the deletion process because DataImportCron CRD is not there:

2022-01-26T15:45:54.154258146Z {"level":"error","ts":1643211954.1541657,"logger":"cdi-operator","msg":"error invoking callback for","type":"*v1.Deployment","error":"the server could not find the requested resource (get dataimportcrons.cdi.kubevirt.io)","stacktrace":"kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/callbacks.(*CallbackDispatcher).InvokeCallbacks
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/callbacks/callback-dispatcher.go:157
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).InvokeCallbacks
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:549
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).InvokeDeleteCallbacks
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:515
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).ReconcileDelete
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:713
kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).Reconcile
	vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:136
kubevirt.io/containerized-data-importer/pkg/operator/controller.(*ReconcileCDI).Reconcile
	pkg/operator/controller/controller.go:233
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
kubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

Version-Release number of selected component (if applicable):
CDI v1.43.1

How reproducible:
100%

Steps to Reproduce:
1. deploy CNV
2. try to install it
3.

Actual results:
CDI CR is marked for deletion but protected by a finalized controlled by CDI operator:

apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
  annotations:
    cdi.kubevirt.io/configAuthority: ""
  creationTimestamp: "2022-01-26T15:24:13Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2022-01-26T15:35:40Z"
  finalizers:
  - operator.cdi.kubevirt.io

DataImportCron CRD is not there anymore

CDI operator is looping forever on:
{"level":"error","ts":1643211954.1541657,"logger":"cdi-operator","msg":"error invoking callback for","type":"*v1.Deployment","error":"the server could not find the requested resource (get dataimportcrons.cdi.kubevirt.io)","stacktrace":"kubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/callbacks.(*CallbackDispatcher).InvokeCallbacks

Expected results:
We can successfully remove the product

Additional info:
I think that let the OLM manage the CRDs is a better and safer idea

Comment 2 Simone Tiraboschi 2022-02-02 08:15:11 UTC
With CDI v1.44.0 which includes https://github.com/kubevirt/containerized-data-importer/pull/2129 now we see a slightly different pattern but basically the bug is still there:

CDI operator is looping forever on

2022-02-01T16:07:40.308086379Z {"level":"error","ts":1643731660.3080544,"logger":"cdi-operator","msg":"error getting apiserver ca bundle","error":"ConfigMap \"cdi-apiserver-signer-bundle\" not found","stacktrace":"kubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.getAPIServerCABundle\n\tpkg/operator/resources/cluster/apiserver.go:542\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createDataImportCronValidatingWebhook\n\tpkg/operator/resources/cluster/apiserver.go:244\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createDynamicAPIServerResources\n\tpkg/operator/resources/cluster/apiserver.go:57\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createResourceGroup\n\tpkg/operator/resources/cluster/factory.go:102\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.createAllResources\n\tpkg/operator/resources/cluster/factory.go:88\nkubevirt.io/containerized-data-importer/pkg/operator/resources/cluster.CreateAllDynamicResources\n\tpkg/operator/resources/cluster/factory.go:77\nkubevirt.io/containerized-data-importer/pkg/operator/controller.(*ReconcileCDI).GetAllResources\n\tpkg/operator/controller/cr-manager.go:126\nkubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).InvokeDeleteCallbacks\n\tvendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:509\nkubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).ReconcileDelete\n\tvendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:713\nkubevirt.io/containerized-data-importer/vendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler.(*Reconciler).Reconcile\n\tvendor/kubevirt.io/controller-lifecycle-operator-sdk/pkg/sdk/reconciler/reconciler.go:136\nkubevirt.io/containerized-data-importer/pkg/operator/controller.(*ReconcileCDI).Reconcile\n\tpkg/operator/controller/controller.go:233\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nkubevirt.io/containerized-data-importer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

Comment 3 Simone Tiraboschi 2022-02-02 08:16:10 UTC
Created attachment 1858560 [details]
CDI operator logs v1.44.0

Comment 4 Natalie Gavrielov 2022-02-03 11:43:07 UTC
Hi Arnon, can you please have a look, is it the same thing (comment 2)

Comment 5 Arnon Gilboa 2022-02-03 15:04:33 UTC
I see no reason to keep both bzs, closing this one as dup.

*** This bug has been marked as a duplicate of bug 2037312 ***


Note You need to log in before you can comment on or make changes to this bug.