Description of problem: HostPathProvisioner CSI custom resource can't be deleted if it's a combination of a basic storage pool, and a pvcTemplate. This causes the hostpath-provisioner-operator to CrashLoopBackOff. We are NOT facing this issue in HPP CR with just a basic storage pool, or in HPP CR with just a pvcTemplate. Version-Release number of selected component (if applicable): 4.10 How reproducible: Always Steps to Reproduce: 1. Get a cnv 4.10 cluster 2. Deploy an HPP CR with both basic storage pool and pvcTemplate 3. Delete HPP CR Actual results: HPP CR is not deleted, hostpath-provisioner-operator - CrashLoopBackOff. Expected results: HPP CR deleted. Additional info: HPP CR YAML: $ cat hpp-cr-basic-pvc-block.yaml apiVersion: hostpathprovisioner.kubevirt.io/v1beta1 kind: HostPathProvisioner metadata: name: hostpath-provisioner spec: imagePullPolicy: IfNotPresent storagePools: - name: local-basic path: "/var/local-basic" - name: local-pvc-template pvcTemplate: volumeMode: Block storageClassName: ocs-storagecluster-ceph-rbd accessModes: - ReadWriteOnce resources: requests: storage: 40Gi path: "/var/hpvolumes" workload: nodeSelector: kubernetes.io/os: linux $ oc delete HostPathProvisioner hostpath-provisioner hostpathprovisioner.hostpathprovisioner.kubevirt.io "hostpath-provisioner" deleted ^C $ $ oc get pods -A | grep hostpath openshift-cnv hostpath-provisioner-csi-2z2jq 4/4 Running 0 8m33s openshift-cnv hostpath-provisioner-csi-cmlxz 4/4 Running 0 8m33s openshift-cnv hostpath-provisioner-csi-tggh6 4/4 Running 0 8m33s openshift-cnv hostpath-provisioner-operator-79c988456c-5mnqf 0/1 CrashLoopBackOff 5 (64s ago) 21m $ oc logs -n openshift-cnv hostpath-provisioner-operator-79c988456c-5mnqf -p {"level":"info","ts":1642427183.4980357,"logger":"cmd","msg":"Go Version: go1.16.6"} {"level":"info","ts":1642427183.498063,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1642427183.4980674,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"} I0117 13:46:24.549470 1 request.go:668] Waited for 1.032877849s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s {"level":"info","ts":1642427187.3572679,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1642427187.3657196,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1642427187.3664596,"logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, admission.Defaulter interface is not implemented","GVK":"hostpathprovisioner.kubevirt.io/v1beta1, Kind=HostPathProvisioner"} {"level":"info","ts":1642427187.366523,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"hostpathprovisioner.kubevirt.io/v1beta1, Kind=HostPathProvisioner","path":"/validate-hostpathprovisioner-kubevirt-io-v1beta1-hostpathprovisioner"} {"level":"info","ts":1642427187.3666127,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-hostpathprovisioner-kubevirt-io-v1beta1-hostpathprovisioner"} {"level":"info","ts":1642427187.3666918,"logger":"cmd","msg":"Starting the Cmd."} I0117 13:46:27.366945 1 leaderelection.go:243] attempting to acquire leader lease openshift-cnv/hostpath-provisioner-operator-lock... {"level":"info","ts":1642427187.366945,"msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1642427187.3670144,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"} {"level":"info","ts":1642427187.3749099,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"} {"level":"info","ts":1642427187.375009,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443} {"level":"info","ts":1642427187.3757684,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"} I0117 13:46:44.468614 1 leaderelection.go:253] successfully acquired lease openshift-cnv/hostpath-provisioner-operator-lock {"level":"info","ts":1642427204.4702606,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703252,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703326,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.470341,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703543,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703605,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703777,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703848,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4703944,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4704046,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4704115,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4704182,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4704297,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4704354,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.4704466,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1642427204.470458,"logger":"controller.hostpathprovisioner-controller","msg":"Starting Controller"} {"level":"info","ts":1642427204.5743701,"logger":"controller.hostpathprovisioner-controller","msg":"Starting workers","worker count":1} {"level":"info","ts":1642427204.574589,"logger":"controller_hostpathprovisioner","msg":"Detected CSI CR, reconciling CSI only","Request.Namespace":"","Request.Name":"hostpath-provisioner"} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x155adb2] goroutine 672 [running]: kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).storagePoolDeploymentByNode(0xc00030c480, 0x1abed68, 0xc001355d80, 0xc00072f600, 0xc00123f6b0, 0xc000046030, 0xd, 0xc0006f2000, 0xc0006f2000) /remote-source/app/pkg/controller/hostpathprovisioner/storagepool.go:330 +0x252 kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).cleanDeployments(0xc00030c480, 0x1abed68, 0xc001355d80, 0xc00072f600, 0xc000046030, 0xd, 0x14, 0x1ad8350) /remote-source/app/pkg/controller/hostpathprovisioner/storagepool.go:116 +0x1ee kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile(0xc00030c480, 0x1ab61f8, 0xc00055b1a0, 0x0, 0x0, 0xc0005b10b0, 0x14, 0xc00055b1a0, 0xc00055b110, 0xc00123fdb0, ...) /remote-source/app/pkg/controller/hostpathprovisioner/controller.go:308 +0x40b sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc0001cab40, 0x1ab61f8, 0xc00055b110, 0x0, 0x0, 0xc0005b10b0, 0x14, 0xc00055b100, 0x0, 0x0, ...) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114 +0x247 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001cab40, 0x1ab6150, 0xc0007bc000, 0x1728440, 0xc0002c8180) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311 +0x305 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001cab40, 0x1ab6150, 0xc0007bc000, 0xc00093a700) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x205 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2(0xc000673c60, 0xc0001cab40, 0x1ab6150, 0xc0007bc000) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x6b created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x425
Verified on CNV v4.10, HCO v4.10.0-595, hostpath-provisioner-operator v4.10.0-56 $ oc get HostPathProvisioner NAME AGE hostpath-provisioner 4s $ oc get pods -A | grep hostpath openshift-cnv hostpath-provisioner-csi-5mdj5 4/4 Running 0 16s openshift-cnv hostpath-provisioner-csi-7hhvf 4/4 Running 0 16s openshift-cnv hostpath-provisioner-csi-wvw65 4/4 Running 0 16s openshift-cnv hostpath-provisioner-operator-64b5dbf66c-9fjnl 1/1 Running 0 3h26m $ oc get pods -A | grep hpp openshift-cnv hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-8sbqc 1/1 Running 0 36s openshift-cnv hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-jcrbh 1/1 Running 0 36s openshift-cnv hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-vgczp 1/1 Running 0 36s $ oc get deployments -A | grep hpp openshift-cnv hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-97lsq 1/1 1 1 75s openshift-cnv hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-szklx 1/1 1 1 75s openshift-cnv hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-z6rkq 1/1 1 1 75s $ oc delete HostPathProvisioner hostpath-provisioner hostpathprovisioner.hostpathprovisioner.kubevirt.io "hostpath-provisioner" deleted $ oc get HostPathProvisioner No resources found $ oc get deployments -A | grep hpp $ oc get pods -A | grep hpp $ oc get pods -A | grep hostpath openshift-cnv hostpath-provisioner-operator-64b5dbf66c-9fjnl 1/1 Running 0 3h28m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947